Display system, network interactive display device, terminal, and control program

ABSTRACT

A display method of a display device connected to a plurality of terminals via a network, the display device displaying one or more split window areas, comprising: receiving identification information and a display request of a terminal among the terminals from the terminal, determining whether or not the terminal is permitted to connect to the display device and whether or not the terminal is permitted to display to the display device, and displaying, when the terminal is a terminal permitted to display, a multi-window screen that is synthesized based on image data and predetermined information received from the terminal.

This is a Divisional of application Ser. No. 12/588,714 filed Oct. 26,2009, which is a Continuation of application Ser. No. 10/623,518 filedJul. 22, 2003. The disclosures of the prior applications are herebyincorporated by reference herein in their entireties.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technique that presents screens of aplurality terminals connected to a network on a single screen of adisplay device in a multi-window presentation fashion.

2. Description of the Related Art

FIG. 25 shows a structure of a conventional display system. Connected toa network 210 as shown are a projector 220, a notebook computer 230, anddesk-top computers 251, 252, 253, and 254 as network interactive displaydevices. Further connected to the network 210 are tablets 261, 262, 263,and 264, each of which directly inputs drawings to the projector 220 andinserts data in the already projected drawings. A screen capturesoftware program is already installed on each of the notebook computer230 and the desk-top computers 251, 252, 253, and 254.

In the conventional display system thus constructed, the screen capturesoftware program captures the content displayed on the screen of thenotebook computer 230, and the captured image data is then sent to theprojector 220 through the network 210. The screen presented on thenotebook computer 230 is thus projected and displayed on the projector220. Operating a remote controller as an attachment to the projector220, the projected image on the projector 220 is switched from a screenof the notebook computer 230 to a screen of the desk-top computer 251,for example.

The image presented on the projector 220 in the conventional displaysystem is one of the computer screen images of the notebook computer230, and the desk-top computers 251, 252, 253, and 254. To compare thecontents of one screen to another of the notebook computer 230 and thedesk-top computers 251, 252, 253, and 254, the user is forced to switchthe screens from one to another. There is a growing need for a functionthat allows screens of a plurality of personal computers on a singlescreen of a display device.

To satisfy such a need, development efforts have been made. However, nosuch a display system satisfying the need exist. For one reason, a largethroughput is required of a projector (a network interactive displaydevice) in a display system, and a workload on a network also increases.

SUMMARY OF THE INVENTION

Accordingly, it is an object of the present invention to provide anetwork interactive display device and a display control program forpresenting display screens of a plurality of terminals, connected to anetwork, on a screen of a display of the display device in amulti-window presentation fashion.

It is also another object of the present invention to provide a displaysystem, a network interactive display device, a display control softwareprogram, a network interactive projector, a network interactive plasmadisplay apparatus, a network interactive liquid-crystal displayapparatus, a terminal, and a control software program, each forpresenting display screens of a plurality of terminals, connected to anetwork, on a screen of a display of the network interactive displaydevice in a multi-window presentation fashion without introducing anincrease in workload on the network interactive display device and thenetwork.

A display system in one aspect of the present invention includes aplurality of terminals, each terminal having a screen capture function,and sending image data, captured using the screen capture function, overa network, and a network interactive display device, including adisplay, and receiving the captured image data transmitted from theterminal through the network, and having a multi-window screenpresentation function for synthesizing the captured image data intosingle screen multi-window format data to be displayed on a displayscreen of the display, wherein, as processes required to present thesingle screen multi-window format data on the display screen of thedisplay of the network interactive display device, the terminal performsa size conversion process of an image size of the image data capturedusing the screen capture function and the network interactive displaydevice acquires the captured image data subsequent to the sue conversionthereof from the terminal, and synthesizes the received captured imagedata.

The size conversion process for a multi-window screen presentationfunction is performed on the terminal. The display system presents thescreens of the plurality terminals connected to the network on thedisplay of the network interactive display device in a multi-windowformat without introducing an increase in workload on the networkinteractive display device and the network.

In the display system of a preferred embodiment of the presentinvention, the network interactive display device may divide the displayscreen of the display into windows of the number equal to the number ofterminals to be displayed, may determine a display size of the windowassigned to each terminal to be displayed, and may send information ofthe display size to the terminal, and the terminal may perform the sizeconversion on the image size of the captured image data to the receiveddisplay size when the terminal receives the display size.

In accordance with the preferred embodiment of the present invention,the terminal performs the size conversion process based on the displaysize determined by the network interactive display device.

In the display system of a preferred embodiment of the presentinvention, the terminal may further perform a color conversion processon the captured image data in accordance with a color count of thedisplay of the network interactive display device before sending thecaptured image data to the network interactive display device, inaddition to the size conversion process on the image data captured usingthe screen capture function.

In the preferred embodiment of the present invention, the workload onthe network interactive display device is further reduced because theterminal performs the color conversion process before sending thecaptured image data to the network interactive display device.

In the display system of a preferred embodiment of the presentinvention, the network interactive display device may also send thecolor count of own display to the terminal when sending the display sizeto the terminal, while the terminal may perform the color conversionprocess in response to the color count received from the networkinteractive display device.

In accordance with the preferred embodiment of the present invention,the terminal performs the color conversion process in response to thecolor count designated by and received from the network interactivedisplay device.

In the display system of a preferred embodiment of the presentinvention, the network interactive display device may include aprojector.

In accordance with the preferred embodiment of the present invention,the network interactive display device is the projector.

In the display system of a preferred embodiment of the presentinvention, the network interactive display device may include a plasmadisplay.

In accordance with the preferred embodiment, a plasma display is used asthe network interactive display device.

In the display system of a preferred embodiment of the presentinvention, the network interactive display device may include aliquid-crystal monitor.

In accordance with the preferred embodiment, a liquid-crystal monitor isused as the network interactive display device.

In the display system of a preferred embodiment of the presentinvention, the network interactive display device may include an organicEL display.

In accordance with the preferred embodiment, an organic EL(Electroluminescent) display is used as the network interactive displaydevice.

In the display system of a preferred embodiment of the presentinvention, the terminal may include one of a personal computer and a PDA(Personal Digital Assistant).

In accordance with the preferred embodiment of the present invention,one of a personal computer and a PDA is used as the terminal.

A network interactive display device in another aspect of the presentinvention is connected to a plurality of terminals through a network,each terminal having a screen capture function, and includes a display,a communication unit for communicating in a two-way fashion with each ofthe terminals, and a display control unit, wherein the communicationunit receives the image data which has been captured by each terminalthrough the screen capture function thereof, and which has been sizeconverted to a predetermined image size by each terminal, and thedisplay control unit has a multi-window screen presentation function forsynthesizing the captured image data received by the communication unitinto single screen multi-window format data to be displayed on a displayscreen of the display.

Since the network interactive display device receives the captured imagedata in the size converted form thereof from the terminal, andsynthesizes the received image data, the workload of processing in themulti-window presentation is reduced in the network interactive displaydevice.

In the network interactive display device of a preferred embodiment ofthe present invention, the display control unit may have an insertionfunction for inserting a new window into a current display screen todisplay the new window.

In accordance with the preferred embodiment, the network interactivedisplay device has the insertion function.

In the network interactive display device of a preferred embodiment ofthe present invention, the user may select at will a terminal, whichprovides the captured image data to be displayed on the display screenof the display, from among the plurality of terminals connected to thenetwork interactive display device.

In accordance with the preferred embodiment, the network interactivedisplay device allows the user to select at will a terminal, which is toprovide the captured image data to be displayed on the display screen ofthe display, from among the plurality of terminals connected to thenetwork interactive display device.

In the network interactive display device of a preferred embodiment ofthe present invention, the terminal that provides the captured imagedata to be displayed on the display screen of she display may beselected in a two-way communication of the communication unit by one ofthe network interactive display device and the terminals.

In accordance with the preferred embodiment, the network interactivedisplay device allows the terminal displaying the captured image data onthe display screen to be selected in a two-way communication of thecommunication unit by one of the network interactive display device andthe terminal.

In the network interactive display device of a preferred embodiment ofthe present invention, the display control unit may have an expansiondisplay function for expanding a predetermined window from among aplurality of windows forming a multi-window screen displayed on thedisplay screen of the display.

In accordance with the preferred embodiment, the network interactivedisplay device has the expansion display function for expanding thepredetermined window.

In the network interactive display device of a preferred embodiment ofthe present invention, the display control unit may have a single-windowscreen selection function for switching the display screen from apredetermined window from among a plurality of windows forming amulti-window screen displayed on the display screen to a single-windowfull screen.

In accordance with the preferred embodiment, the network interactivedisplay device has the single-window screen selection function forswitching the display screen from the predetermined window to thesingle-window full screen.

In the network interactive display device of a preferred embodiment ofthe present invention, the display control unit may have an erasefunction for erasing a predetermined window from among a plurality ofwindows forming a multi-window screen displayed on the display screen ofthe display.

In accordance with the preferred embodiment, the network interactivedisplay device has the erase function for erasing the predeterminedwindow.

In the network interactive display device of a preferred embodiment ofthe present invention, the predetermined window may be selected inresponse to an operation by the user.

In accordance with preferred embodiments, the network interactivedisplay device allows the user to select the window to be expanded, thewindow to be switched to the single-window full screen, and the windowto be erased.

In the network interactive display device of a preferred embodiment ofthe present invention, the predetermined window may be selected by oneof the network interactive display device and the terminal in a two-waycommunication of the communication unit thereof.

In the preferred embodiment, the window to be expanded, the window to beswitched to the single-window full screen, or the window to be erased isdesignated by one of the network interactive display device or theterminal.

In the network interactive display device of a preferred embodiment ofthe present invention, the image captured data received from theterminal may be obtained by designating the whole or a portion of thedisplay screen of the terminal.

In accordance with the preferred embodiment, the network interactivedisplay device displays the captured image data, which is obtained usingfull-screen capturing or partial-screen capturing.

In the network interactive display device of a preferred embodiment ofthe present invention, the captured image data received from theterminal may be obtained by detecting and capturing only a change on thedisplay screen of the terminal.

In accordance with the preferred embodiment, the workload on the networkis reduced by capturing only the change on the screen of the terminal.The network interactive display device thus presents image data which isa combination of existing captured data and the changed component ofdata.

In a preferred embodiment of the present invention, the networkinteractive display device may include a display size determining unitthat divides the display screen of the display into windows of thenumber equal to the number of terminals to be displayed, and determinesa display size of the window to which the terminal to be displayed isassigned, and a controller that sends the display size determined by thedisplay size determining unit to the corresponding terminal through thecommunication unit, wherein the controller receives, through thecommunication unit, the captured image data, having the converted sizeequal to the display size of the window assigned to the terminal, fromthe terminal to which the display size is sent, and controls the displaycontrol unit to synthesize the received captured image data into singlescreen multi-window format data to be displayed on the display screen ofthe display.

In accordance with the preferred embodiment, the display size convertedby the terminal is set to be display size determined by the display sizedetermining unit.

In the network interactive display device of a preferred embodiment ofthe present invention, an aspect ratio of the window assigned to theterminal to be displayed may be equalized to an aspect ratio of thedisplay screen of the display of the terminal.

In accordance with the preferred embodiments of the present invention,the network interactive display device provides a display screen freefrom discordance.

In the network interactive display device of a preferred embodiment ofthe present invention, through the communication unit, the controllermay also send a display color count of the display to the terminal whensending the display size to the terminal, may receive the captured imagedata having the converted size equal to the display size of the windowassigned to the terminal and having the display color count converted tothe display color count of the display of the network interactivedisplay device, from the terminal to which the display size and thedisplay color count have been sent, and may control the display controlunit to synthesize the received captured image data into single screenmulti-window format data to be displayed on the display screen of thedisplay.

In accordance with the preferred embodiment, the network interactivedisplay device receives, from the terminal, the captured image data theterminal has color converted in addition to the size conversion processfor contraction and synthesizes the received image data. The workload ofprocessing in the multi-window presentation is reduced in the networkinteractive display device.

In the network interactive display device of a preferred embodiment ofthe present invention, a communication protocol of the communicationunit may include the TCP/IP protocol.

In accordance with the preferred embodiment, the widely used TCP/IP isused as the communication protocol of the communication unit.

in the network interactive display device of a preferred embodiment ofthe present invention, the network may include one of a LAN, a radioLAN, and a near-field communication radio LAN.

In accordance with the preferred embodiment, one of the LAN (Local-AreaNetwork), the radio LAN, the near-field communication radio LAN is usedas the network.

A network interactive projector in yet another aspect of the presentinvention includes one of the above-referenced network interactivedisplay devices.

In accordance with the above aspect, the projector has theabove-referenced advantages of the network interactive display device.

In the network interactive projector of a preferred embodiment of thepresent invention, the display may include one of a liquid-crystal lightvalve, an LCoS light valve, and a DMD (Digital Micromirror Device)(Trademark of Texas Instruments).

In accordance with the preferred embodiment, the projector including oneof the liquid-crystal light valve, the LCoS light valve, and the DMD hasthe above-referenced advantages of the network interactive displaydevice.

A network interactive plasma display apparatus in yet another aspect ofthe present invention includes one of the above-referenced networkinteractive display devices, wherein the display includes a plasmadisplay panel.

In accordance with the above aspect of the present invention, the plasmadisplay apparatus provides the above-referenced advantages of thenetwork interactive display device.

A network interactive liquid-crystal display apparatus in yet anotheraspect of the present invention includes one of the above-referencednetwork interactive display devices, wherein the display includes aliquid-crystal panel.

In accordance with the above aspect of the present invention, theliquid-crystal display apparatus have the above-referenced embodimentsof the network interactive display device.

A network interactive organic EL display apparatus in a further aspectof the present invention includes one of the above-referenced networkinteractive display devices, wherein the display includes an organic ELpanel.

In accordance with the above aspect of the present invention, thenetwork interactive organic EL display apparatus provides theabove-referenced advantages of the network interactive display device.

The present invention in a further aspect relates to a display controlsoftware program of a CPU that constitutes the display control unit ofone of the above-referenced network interactive display devices.

In accordance with the above aspect of the present invention, thedisplay control program allows a display device to provides theabove-referenced advantages of the network interactive display device.

In yet another aspect of the present invention, a terminal, connected toone of the above-referenced network interactive display devices includesa display, a communication unit that communicates in a two-way fashionwith the network interactive display device, a screen capture processorthat captures the content displayed on the display screen of thedisplay, an image converter which converts the image data captured bythe screen capture processor to image data having a predetermined imagesize, and a controller that sends the captured image data, sizeconverted by the image converter, from the communication unit to thenetwork interactive display device, wherein the terminal generates thecaptured image data that is to be displayed on one of the multi windowsdisplayed on the display screen of the network interactive displaydevice.

When the captured image data to be displayed on one of the windows ofthe screen of the network interactive display device is generated, apart of the process required for the multi-window presentation, i.e.,the size conversion process is performed by the terminal. The terminalthus contributes to a reduction in the workload on the networkinteractive display device.

In the terminal of a preferred embodiment of the present invention, thedisplay screen of the display of the network interactive display devicemay be divided into windows of the number equal to the number ofterminals to be displayed, a display size of the window assigned to eachterminal to be displayed is determined, and the image converter convertsthe image data captured by the screen capture processor to image datahaving the display size assigned to own terminal.

In accordance with the preferred embodiment, the size conversion processis carried out based on the display size determined by the networkinteractive display device.

In the terminal of a preferred embodiment of the present invention, theimage converter may perform a color conversion on the captured imagedata to match the display color count of the display of the networkinteractive display device in addition to the size conversion process,and the controller may send the captured image data, which has beensubjected to the size conversion process and the color conversionprocess, from the communication unit to the network interactive displaydevice.

In accordance with the preferred embodiment, the image converterperforms the color conversion on the captured image data in addition tothe size conversion process, thereby further reducing the workload onthe network interactive display device.

The present invention in a further aspect relates to a control softwareprogram of a CPU that constitutes each processor of one of theabove-referenced terminals.

In accordance with the above aspect, the control program allows aterminal to have the above-referenced advantages of the above terminal.

The present invention in a further aspect relates to a networkinteractive display device connected to each of a plurality of terminalsthrough a network, each terminal having a screen capture function, andincludes a display, a communication unit for communicating in a two-wayfashion with each of the terminals, and a display control unit, whereinthe display control unit has a multi-window screen presentation functionfor synthesizing the captured image data, captured by each terminalthrough the screen capture function and received by the communicationunit, into single screen multi-window format data to be displayed on adisplay screen of the display.

The network interactive display device presents the screens of theplurality terminals connected to the network on the display screen of adisplay of the network interactive display device in a multi-windowformat.

In the network interactive display device of a preferred embodiment ofthe present invention, the display control unit may have an insertionfunction for inserting a new window into a current display screen todisplay the new window.

In accordance with the preferred embodiment, the network interactivedisplay device has the insertion function.

In the network interactive display device of a preferred embodiment ofthe present invention, the user may select at will a terminal, whichprovides the captured image data to be displayed en the display screenof the display, from among the plurality of terminals connected to thenetwork interactive display device.

In accordance with the preferred embodiment, the network interactivedisplay device allows the user to select at will a terminal, which is toprovide the captured image data to be displayed on the display screen ofthe display, from among the plurality of terminals connected to thenetwork interactive display device.

In the network interactive display device of a preferred embodiment ofthe present invention, the terminal that provides the captured imagedata to be displayed on the display screen of the display may beselected in a two-way communication of the communication unit by one ofthe network interactive display device and the terminal.

In accordance with the preferred embodiment, the network interactivedisplay device allows the terminal providing the captured image data onthe display screen to be selected in a two-way communication of thecommunication unit by one of the network interactive display device andthe terminal.

In the network interactive display device of a preferred embodiment ofthe present invention, the display control unit may have an expansiondisplay function for expanding a predetermined window from among aplurality of windows forming a multi-window screen displayed on thedisplay screen of the display.

In accordance with the preferred embodiment, the network interactivedisplay device has the expansion display function for expanding thepredetermined window.

In the network interactive display device of a preferred embodiment ofthe present invention, the display control unit may have a single-windowscreen selection function for switching the display screen from apredetermined window from among a plurality of windows forming amulti-window screen displayed on the display screen of the display to asingle-window full screen.

In accordance with the preferred embodiment, the network interactivedisplay device has the single-window screen selection function forswitching the display screen from the predetermined window to thesingle-window full screen.

In the network interactive display device of a preferred embodiment ofthe present invention, the display control unit may have an erasefunction for erasing a predetermined window from among a plurality ofwindows forming a multi-window screen displayed on the display screen ofthe display.

In accordance with the preferred embodiment, the network interactivedisplay device has the erase function for erasing the predeterminedwindow.

In the network interactive display device of a preferred embodiment ofthe present invention, the predetermined window may be selected inresponse to an operation by the user.

In accordance with preferred embodiments, the network interactivedisplay devise allows the user to select the window to be expanded, thewindow to be switched to the single-window full screen, and the windowto be erased.

In the network interactive display device of a preferred embodiment ofthe present invention, the predetermined window may be selected by oneof the network interactive display device and the terminal in a two-waycommunication of the communication unit thereof.

In the preferred embodiment, any of the window to be expanded, thewindow to be switched to the full-screen mode, or the window to beerased is designated by one of the network interactive display deviceand the terminal.

In the network interactive display device of a preferred embodiment ofthe present invention, the captured image data received from theterminal may be obtained by designating the whole or a portion of thedisplay screen of the terminal.

In accordance with the preferred embodiment, the network interactivedisplay device displays the captured image data, which is obtained usingfull-screen capturing or partial-screen capturing.

In the network interactive display device of a preferred embodiment ofthe present invention, the captured image data received from theterminal may be obtained by detecting and capturing only a change on thedisplay screen of the terminal.

In accordance with the preferred embodiment, the workload on the networkis reduced by capturing only the change on the screen of the terminal.The network interactive display device thus presents image data which isa combination of existing captured data and the changed component ofdata.

In the network interactive display device of a preferred embodiment ofthe present invention, the display control unit may include a windowarea information generator which divides the display screen of thedisplay into windows of the number equal to the number of terminals tobe displayed, and generates window area information containing a displaysize of the window to which the terminal to be displayed is assigned,and information identifying a display position of the window, an imagesynthesizer which synthesizes the captured image data from the terminalsinto single screen multi-window format data in accordance with thewindow area information generated by the window area informationgenerator, thereby generating synthesized image data, and an imageprocessor which processes the synthesized image data generated by theimage synthesizer, thereby generating display image data and outputtingthe display image data to the display.

In the network interactive display device of a preferred embodiment ofthe present invention, the image synthesizer may synthesize the capturedimage data by contracting or expanding the captured image data from eachterminal with an aspect ratio of the image size of the captured imagedata maintained.

In accordance with the preferred embodiment, the network interactivedisplay device provides a display screen free from discordance.

In the network interactive display device of a preferred embodiment ofthe present invention, a communication protocol of the communicationunit may include the TCP/IP protocol.

In accordance with the preferred embodiment, the widely used TCP/IP isused as the communication protocol of the communication unit.

In the network interactive display device of a preferred embodiment ofthe present invention, the network may include one of a LAN, a radioLAN, and a near-field communication radio LAN.

In accordance with the preferred embodiment, one of the LAN, the radioLAN, the near-field communication radio LAN is used as the network.

A network interactive projector in a further aspect of the presentinvention includes one of the above-referenced network interactivedisplay devices.

In accordance with the above aspect of the present invention, theprojector has the above-referenced advantages of the network interactivedisplay device.

In the network interactive projector of a preferred embodiment of thepresent invention, the display may include one of a liquid-crystal lightvalve, an LCoS light valve, and a DMD.

In accordance with the preferred embodiment, the projector including oneof the liquid-crystal light valve, the LCoS light valve, and the DMD hasthe above-referenced advantages of the network interactive displaydevice.

A network interactive plasma display apparatus in a further aspect ofthe present invention includes one of the above-referenced networkinteractive display devices, wherein the display includes a plasmadisplay panel.

In accordance with the above aspect, the plasma display apparatus havethe above-referenced advantages of the network interactive displaydevice.

A network interactive liquid-crystal display apparatus in a furtheraspect of the present invention includes one of the above-referencednetwork interactive display devices, wherein the display includes aliquid-crystal panel.

In accordance with the above aspect, the liquid-crystal displayapparatus have the above-referenced embodiments of the networkinteractive display device.

A network interactive organic EL display apparatus in a further aspectof the present invention includes one of the above-referenced networkinteractive display devices, wherein the display includes an organic ELpanel.

In accordance with the above aspect, the network interactive organic ELdisplay apparatus provides the above-referenced advantages of thenetwork interactive display device.

The present invention in a further aspect relates to a display controlsoftware program of a CPU that constitutes the display control unit ofone of the above-referenced network interactive display devices.

In accordance with the above aspect, the display control programprovides a display device with the above-referenced advantages of one ofthe above network interactive display devices.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a network of a display system including

a network interactive display device in accordance with preferredembodiments of one present invention;

FIG. 2 is a block diagram illustrating the structure of a terminal inaccordance with a first preferred embodiment of the present invention;

FIG. 3 is a block diagram illustrating the structure of a networkinteractive display device of the first preferred embodiment;

FIG. 4 is a functional diagram illustrating the function of the networkinteractive display device of the first preferred embodiment;

FIG. 5 illustrates items of terminal information managed in a displaystatus management file;

FIG. 6 is a flow diagram illustrating the operation of the firstpreferred embodiment;

FIG. 7 is a continuation of the flow diagram of FIG. 6;

FIG. 8 illustrates the configuration of the display system in whichdisplay screens of four terminals 1 a-1 d are presented on a displayscreen of a display device;

FIG. 9 is a flow diagram illustrating the flow of a window areainformation generation process;

FIG. 10 illustrates a specific structure of the display statusmanagement file;

FIG. 11A illustrates a specific structure of a table held in a tentativewindow area setting file, and FIG. 11B illustrates a tentative windowarea based, on the table of FIG. 11A;

FIG. 12 illustrates a true window area size and a true origin;

FIG. 13 illustrates one example of a window area information file;

FIG. 14 is a flow diagram of an operation of the terminal which hasreceived a capture start command and a display status management filefrom the display device of the first preferred embodiment;

FIG. 15 diagrammatically illustrates the display system to explain anexpansion display function;

FIG. 16A illustrates a tentative window area setting table which isreferenced when a priority order is updated, and FIG. 16B illustrates atentative window area based on the table of FIG. 16A;

FIG. 17 illustrates a true window area with the priority order modified;

FIG. 18 illustrates a window area information file that is produced whenthe priority order is modified;

FIG. 19 diagrammatically illustrates a single-window screen presentationfunction;

FIG. 20 diagrammatically illustrates an insertion function;

FIG. 21 diagrammatically illustrates a window erase function;

FIG. 22 is a flow diagram illustrating an operation of terminal whichhas received a difference capture start command and a display statusmanagement file from the display device;

FIG. 23 illustrates a screen comparison process wherein a mouse pointeris moved;

FIG. 24 illustrates the screen comparison process of FIG. 23;

FIG. 25 illustrates a conventional display system;

FIG. 26 is a block diagram illustrating a structure of the terminal inaccordance with a second preferred embodiment of the present invention;

FIG. 27 is a block diagram illustrating a structure of the networkinteractive display device of the second preferred embodiment;

FIG. 28 is a functional diagram of an operation of the networkinteractive display device of the second preferred embodiment of thepresent invention;

FIG. 29 is a flow diagram illustrating an operation of the displaysystem of the second preferred embodiment of the present invention; and

FIG. 30 is a flow diagram illustrating the operation of the terminalwhich has received a capture start command and a display statusmanagement file from the display device of the second preferredembodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS First Embodiment

FIG. 1 illustrates a network of a display system 100 including a networkinteractive display device 2 in accordance with preferred embodiments ofthe present invention.

The display system 100 includes a plurality of terminals (only fourterminals 1 a, 1 b, 1 c, and 1 d are shown in FIG. 1), and the networkinteractive display device 2 (a projector here) having a multi-windowscreen presentation function as one of major functions of the presentinvention. The plurality of terminals 1 are respectively connected tothe network interactive display device 2 (hereinafter simply referred tothe display device 2) through a network 3 in a two-way communicationbased on the TCP/IP protocol. A unique name is provided beforehand toeach terminal 1 (hereinafter referred to as a terminal name). Thenetwork 3 may be any of a LAN (Local Area Network), a radio LAN, and anear-field communication radio LAN such as Bluetooth (Tradename ofBluetooth SIG Inc., U.S.A.).

The display system 100 allows screens presented on the plurality ofterminals 1 to be concurrently presented on a multi-window displayscreen of the display device 2. Such a system 100 is useful in aconference or a presentation. The terminal 1 and the display device 2will now be discussed in detail.

FIG. 2 is a block diagram illustrating the structure of the terminal 1in accordance with a first preferred embodiment of the presentinvention.

The terminal 1 may be a personal computer or a PDA (Personal DigitalAssistant). The terminal 1 includes a display 11 for presenting adiversity of information such as materials for presentation, a videomemory 12 for storing the content to be presented on the display 11, aninput section 13 including of a tablet, a mouse, or a keyboard, a userinterface 14 for detecting an operational input entered from the inputsection 13 and outputting the operational input to a controller (CPU)16, a storage 15 for storing application software programs (such as acontrol program) for performing the processes of the present inventionand a variety of pieces of data, the controller 16, and a communicationunit 17.

The control program stored in the storage 15 is used to perform aterminal control function to perform a multi-window screen presentationfunction on the network interactive display device 2, a screen capturefunction to capture a whole or a part of the screen of the display 11,an image conversion function to convert captured image data acquiredusing the screen capture function into data in a format of a display 21of the network interactive display device 2, and a function to detect achange on the screen of the display 11. The application softwareprograms and the CPU constitute a data management processor 18, a screencapture processor 19, an image converter 19A, and a screen comparisonprocessor 20.

The controller 16 receives a variety of requests, including a connectionrequest, a display request, an expansion display request, a request toswitch to a single-window full screen, and an erase request through theuser interface 14 or the communication unit 17, and performs processesresponsive to each request. Under the control of the controller 16, theimage converter 19A converts the image data acquired by the screencapture processor 19, and the communication unit 17 sends the convertedcaptured image data to the network interactive display device 2.

The communication unit 17 carries out a two-way communication with thenetwork interactive display device 2. The communication protocol usedhere is the TCP/IP. The communication unit 17 has a protocol processingfunction for ARP, ICMP, IP, TCP, UDP, etc. required for the TCP/IPconnection. This protocol processing function is carried out under thecontrol of an OS.

The conversion processes performed by the image converter 19A arerequired when the display device 2 to be discussed later performs amulti-window screen presentation. Specifically, the conversion processesinclude a size conversion process to convert the captured image datainto data in a display size of a window assigned to own terminal 1, anda color conversion process to convert the captured image data into datahaving a display color count of the display 21 of the display device 2.The terminal 1 performs the conversion processes, required to present amulti-window screen on the display device 2, on the captured image dataacquired by the screen capture processor 19, and then sends sheconverted captured image data to the display device 2.

FIG. 3 is a block diagram illustrating the structure of the displaydevice 2 of the first preferred embodiment.

The display device 2 includes the display 21, a display control unit 22which has a multi-window screen presentation function, an expansiondisplay function, a function to switch to the single-window full screen,an insertion function, and an erase function, and controls the displayscreen to be presented on the display 21, an input section 23 includinga remote controller, a mouse, or a keyboard, a user interface 24 fordetecting an operational input from the input section 23 and foroutputting the operational input to a controller 27 to be discussedlater, a program storage 25 for storing the display control program toperform the multi-window screen presentation function of the presentinvention, a data storage 26 for storing a variety of files and datarequired to carry out the display control program, a controller (CPU) 27for generally controlling the display device 2, and a communication unit28 for performing a two-way communication with each terminal 1.

The communication unit 28 carries out a two-way communication with theterminal 1. The communication protocol used here is the TCP/IP. Thecommunication unit 28 has a protocol processing function for ARPICMP,IP, TCP, UDP, etc. required for the TCP/IP connection.

The display device 2 may be a plasma display, or a liquid-crystaldisplay instead of the projector shown in FIG. 1. FIG. 3 shows majorportions only related to the context of the present invention, and doesnot show other elements respectively uniquely relating to the projector,the plasma display, and the liquid-crystal display because they are notclosely related to the context of the present invention. If the elementsshown in FIG. 3 are added to each of an existing projector, an existingplasma display, and an existing liquid-crystal display, theyrespectively become a network interactive projector, a networkinteractive plasma display, and a network interactive liquid-crystaldisplay. The display 21 is different depending on the type of thedisplay device 2. Specifically, the display 21 is one of aliquid-crystal light valve, an LCoS light valve, or a DMD (DigitalMicromirror Device) (Trademark of Texas Instruments) in the projector.The display screen of the display 21 becomes a projecting screen. Thedisplay 21 is a plasma display panel in a plasma display device, aliquid-crystal panel in a liquid-crystal display device, or an organicEL (Electroluminescent) panel in an organic EL display device.

Referring to FIG. 4, a variety of files stored in the data storage 26 isdiscussed.

The data storage 26 stores a display specification management file 30, apermitted connection management file 31, a permitted display managementfile 32, a connection status management file 33, a display statusmanagement file 34, a tentative window area setting file 35, and awindow area information file 36. The data storage 26 further includes acaptured image data memory 37 for storing the captured image data sentfrom each terminal 1.

The display specification management file 30 registers a screen sizerepresenting the number of pixels in vertical and horizontal directionsof the display screen of the display 21, and color count informationrepresenting a display color count of the display 21. In this preferredembodiment, the screen size is 1280×1024 (SXGA), and the color count is167,777,216 colors.

The permitted connection management file 31 registers a terminal name ofa terminal 1 which is permitted for connection. The permitted displaymanagement file 32 registers a terminal name of a terminal 1 which ispermitted for screen display.

The connection status management file 33 registers a terminal name of aterminal 1 which is currently connected to the display device 2.

The display status management file 34 manages a display status of thecurrent display 21. The display status management file 34 manages, in atable form, terminal information relating to the terminal 1 that is asource of the captured image data currently presented on the displayscreen of the display 21. The display status management file 34 isupdated each time the display screen of the display 21 is modified. Forexample, if the display screen is switched from a four-window screen toa three-window screen, terminal information of the terminal 1corresponding to an erased window is deleted. If the display screen isswitched from a four-window screen to a five-window screen, terminalinformation of the terminal 1 corresponding to an added window is newlyregistered.

FIG. 5 illustrates items of the terminal information managed in thedisplay status management file 34.

The display status management file 34 contains, as items thereof, a“terminal name”, an “IP address”, a “screen size”, “color countinformation”, “priority”, a “capture area management flag”, a“difference capture management flag”, a “captured image size”, a“difference captured image size 1”, a “difference captured image size2”, a “difference capture origin 1”, and a “difference capture origin2”.

The “terminal name” is a name provided beforehand to the terminal 1. The“screen size” is the number of pixels in the vertical and horizontaldirections of the display screen of the display 11. For example, an SXGAterminal has 1280×1024 pixels, and an XGA terminal has 1024×768 pixels.The “color count information” represents the number of display colors ofthe display 11, and may be 256 colors, or 167,777,216 colors, forexample. The terminal name, the IP address, the screen size, and thecolor count information are the items that must be stored in the displaystatus management file 34 during a registration. Other items are set(updated) by the user as necessary.

The “priority” determines the size of the display size of each windowassigned to the terminal 1 that is identified by the terminal name. Thepriority takes “highest”, “high” or “none”. As will be discussed indetail, the display size having a higher priority order becomes larger.The “capture area management flag” manages the capturing as to whetherthe screen of the terminal 1 identified by the terminal name is capturedin a full-screen capture mode or a partial-screen capture mode. Thecapture area management flag is “0” in the full-screen capture mode,which is a standard capture mode, and “1” in the partial-screen capturemode.

The “difference capture management flag” manages the capture of whetherthe screen of the terminal 1 identified by the terminal name is capturedin a normal capture mode or a change capture mode (hereinafterreferenced to as a difference capture mode). The difference capturemanagement file is “0” in the normal capture mode, or “1” In thedifference capture mode.

The “captured image size” is the size of the captured image data (thenumber of pixels in the vertical and horizontal directions) when thecapture area management flag is “1”, i.e., in the partial-screen capturemode.

The “difference captured image size 1” and the “difference capturedimage size 2” represent the sizes of two different areas acquired in thedifference capture when the difference capture management flag is “1”.The “difference capture origin 1”, and the “difference capture origin 2”are the origins of the two different areas acquired in the differencecapture, and are the absolute coordinates within an area defined by thecaptured image size.

The tentative window area setting file 35 is a file in which informationidentifying a tentative window area assigned to the terminal 1 is setbeforehand. The tentative window area setting file 35 contains aplurality of tables, each table prepared for the terminals. The tablehas a structure as shown in FIG. 11A and FIG. 16A. The table will bediscussed in more detail later. The window area information file 36 willalso be discussed later.

When a predetermined operation is performed on the input section 23, thedisplay 21 displays the contents of the files 30, 31, 32, 33, 34, 35,and 36. The user thus checks and modifies the data on the display screenat will.

Returning to FIG. 4, the display control unit 22 includes a window areainformation generator 41 as a display size determining unit, an imagesynthesiser 42, and an image processor 43. The controller 27 receives avariety of requests such as a connection request, a display request, aninsertion display request, an erase request, etc., received through theuser interface 24 or the communication unit 28. In response to theserequests, under the control of the controller 27, the processors 41, 42,and 43 respectively perform required processes while accessing necessaryfiles in the data storage 26. The controller 27 thus controls thedisplay 21. A display control program, stored in the program storage 25,for providing a multi-window screen presentation function and thecontroller (CPU) 27 constitute the display control unit 22.

From the display status management file 34, the window area informationgenerator 41 learns the number of terminals 1 to be presented, and thepriority order and the screen size of each terminal 1. The window areainformation generator 41 splits the display screen of the display 21 inaccordance with the number of terminals 1 to be presented, and thepriority order and the screen size of each terminal 1. The window areainformation generator 41 generates the window area informationcontaining display size (hereinafter referred to as a window area size)of the window on the display 21 assigned to each terminal 1 to bedisplayed, and information identifying the display position of thewindow (the absolute coordinates at the top left corner of the windowwith respect to the display screen, hereinafter also referred to as anorigin). The Information is stored in the data storage 26 as the windowarea information file 36.

The controller 27 sends, to each terminal 1 to be displayed from thecommunication unit 28, the window area size of the window assigned toeach terminal 1 to be displayed in the window area information file 36generated in the window area information generator 41 together with thedisplay color count of the display 21 held in the display specificationmanagement file 30. The captured image data memory 37 then stores thecaptured image data returned from each terminal 1 that has receivedthese pieces of information, namely, the captured image data that hasbeen subjected to the size conversion process and the color conversionprocess in accordance with the received window area size and displaycolor count. The information referred to as the window area size isidentical to a true window area size in the discussion that follows.

The image synthesizer 42 synthesizes the size converted and colorconverted captured image data from each terminal 1 stored in thecaptured image data memory 37 in accordance with the window areainformation file 36 generated by the window area information generator41. Synthesized image data thus results.

The image processor 43 performs a scanning frequency conversion processon a variety of pieces of image data such as the synthesized image datagenerated by the image synthesizer 42, and the display status managementfile 34 of the data storage 26 which is referenced using an OSD(on-screen display) function, thereby generating display image data andoutputting the display image data to the display 21. The image processor43 includes a scan converter, for example.

The operation of the first preferred embodiment of the present inventionwill now be discussed. FIGS. 6 and 7 are flow diagrams illustrating theoperation of the first preferred embodiment.

A predetermined operational input is entered in the input section 23 inthe display device 2 in a preliminary step for multi-window screenpresentation. Upon detecting the operational input through the userinterface 24, the controller 27 broadcasts a request to return aterminal name and an IP address together with the IP address of thedisplay device 2 through the communication unit 28 and the network 3.When each terminal 1 receives the broadcast request to return theterminal name and the IP address, the terminal 1 returns own terminalname and IP address to the display device 2.

The display device 2 receives a reply (the terminal name and the IPaddress) from each terminal 1 through the communication unit 28, anddetermines whether each terminal is a connection permitted terminal.Specifically, the display device 2 determines whether the returnedterminal name agrees with a terminal name registered in the permittedconnection management file 31. If it is determined that the returnedterminal name agrees with the registered terminal name, the displaydevice 2 handles the terminal 1 as a connection permitted terminal.

The terminal names and the IP addresses of the terminals 1 determined asconnection permitted terminals are successively registered in theconnection status management file 33. The connection status managementfile 33 allows the display device 2 to learn how many terminals 1 arecurrently connected. Since the determination of whether the connectionis permitted or not is based on the terminal name, the system works evenif the IP address, provided to the terminal 1 using the DHCP, becomesdifferent each time connection is made.

The display device 2 waits on standby for any request after the abovepreliminary step 1 s complete. For example, the display screens of thefour terminals 1 a-1 d, out of the terminals 1 operated by conferenceparticipants, are presented on a multi-window display screen 50 of thedisplay device 2. As for the resolutions thereof, the terminal 1 a hasan SXGA resolution (1280×1024 pixels), the terminal 1 b has an SVGAresolution (800×600 pixels), the terminal 1 c has an XGA resolution(1024×768 pixels), and the terminal 1 d has a resolution of 480×640pixels.

Multi-Window Screen Presentation Function

FIG. 8 illustrates the configuration of the display system in whichdisplay screens of four terminals 1 a-1 d are presented on the displayscreen of the display device 2.

The user operates the remote controller in the input section 23 to enteran operational input to display the screens of the terminals 1 a-1 d.Through the user interface 24, the controller 27 is notified of theinput information, namely, a request to display the screens of theterminals 1 a-1 d together with identification information of theterminals 1 a-1 d (step S1). When the request is placed, the priorityorder, the partial capture, and the difference capture may also bedesignated. Here, no particular designation is performed.

Upon receiving the display request, the controller 27 in the displaydevice 2 performs processes in steps S3-S9 for each of the terminals 1a-1 d to be displayed (step S2). More specifically, the controller 27references the permitted connection management file 31 and the permitteddisplay management file 32 according to the terminal name indicated bythe identification information contained in the display request, therebydetermining whether or not each terminal 1 is permitted for connectionand whether or not each terminal 1 is permitted for display (step S3).If it is determined that each terminal 1 is permitted for bothconnection and display (step S4), the controller 27 requests, throughthe communication unit 28, each terminal 1 to send the terminalinformation (the terminal name, the IP address, the screen size, and thecolor count information) (step S3). The controller 27 receives theterminal information which has been sent in response to the request(step S6), and registers the terminal information in the display statusmanagement file 34 (step S7). If the priority order, the partialcapture, and the difference capture are designated during the placementof the display request, the priority order, the capture area managementflag, and the difference capture management flag are also registered inthe registration in step S7.

The controller 27 sends the display status management file 34 and ascreen capture start command to the terminal 1 which is permitted forconnection and display (step S8). When any terminal 1 is not permittedfor connection and display, the controller 27 sends a notification tothat effect to the terminal 1 (step S9).

If all four terminals 1 a-1 d are permitted for connection and display,the terminal information from the terminals 1 a-1 d is registered in thedisplay status management file 34 in the processes in steps S2-S9. Atthe same time, the screen capture start command is sent together withthe display status management file 34 to each of the terminals 1 a-1 dthrough the communication unit 28.

Subsequent to the above processes, the controller 27 notifies the windowarea information generator 41 in the display control unit 22 of a windowarea split request. The display device 2 then enters a window areainformation generation process (step S10). The operation of the terminal1 having received the screen capture start command will be discussedlater. Discussed first is the window area information generation processperformed by the window area information generator 41 in response to thewindow area split request.

FIG. 9 is a flow diagram illustrating the flow of a window areainformation generation process. The operation of the window areainformation generator 41 is specifically discussed on the assumptionthat the display status management file 34 is constructed as shown inFIG. 10. As shown in FIG. 10, terminal names PC-1, PC-2, PC-3, and PDA-1correspond to the terminals 1 a, 1 b, 1 c and 1 d, respectively.

Upon receiving the window area split request from the controller 27, thewindow area information generator 41 learns the number of the terminals1 to be displayed (here, four terminals 1) referencing the displaystatus management file 34. The window area information generator 41 alsolearns the priority order of each of the terminals 1 a-1 d (step S21).The window area information generator 41 references the tentative windowarea setting file 35 according to the number of terminals 1 and thepriority order of each of the terminals 1 a-1 d, and acquires atentative size and a tentative origin of each tentative window areaassigned to each of the terminals 1 a, 1 b, 1 c, and 1 d (step S22). Aswill be clarified later, the adjective “tentative” is used because thearea window here assigned to the terminal 1 is updated in a later stepto size convert the captured image data.

As shown in FIG. 10, the four terminals 1 are to be displayed here andno priority order is set to all of the four terminals 1. A tentativearea setting table in the tentative window area setting file 35 isorganized as shown in FIG. 11A. Here, the display screen of the display21 has a resolution of 1280×1024 (SXGA), and the tentative area settingtable shown in FIG. 11A is organized based on this display screen. FIG.11B shows the tentative window area based on the table window areasetting table shown in FIG. 11A.

The priority order shown in FIG. 11A is determined based on the“priority” item in the display status management file 34, and theterminals 1 are first, second, third, and fourth from the high order tothe low order. The terminals 1 are assigned the “tentative size” and the“tentative origin” for the tentative window area in the lower table. Allthe terminals 1 a-1 d have “none” in the priority order row with nopriority order set therefor (see FIG. 10). If no priority order is set,the order of assignment may be a predetermined one, or may be the orderof registration to the display status management file 34. In the firstpreferred embodiment, the terminals 1 a, 1 b, 1 c, and 1 d (hereinafterreferred to as the terminal names PC-1, PC-2, PC-3, and PDA-1 asappropriate) are assigned tentative window areas 50A, 50B, 50C, and 50Din that order.

The window area information generator 41 further acquires the screensizes of the PC-1, PC-2, PC-3, and PDA-1 from the display statusmanagement file 34 (see FIG. 10), and determines the sizes and originsof true windows respectively assigned thereto based on the acquiredscreen sizes (step S23).

FIG. 12 illustrates the true window area size and the true origin. Asshown, 51A, 51B, 51C, and 51D represent the true window areas assignedto the PC-1, PC-2, PC-3, and PDA-1, respectively. The captured imagedata to be displayed on the tentative window area is size converted withthe aspect ratio thereof maintained. The true window areas are displayareas of the tentative window areas 50A, 50B, 50C, and 50D in which theconverted images are respectively displayed with the centers thereofaligned to be centered on the respective tentative window areas. ThePC-2 is now specifically discussed in connection with the true windowarea size to determine the true window area. The screen size of the PC-2is 1024×768 pixels (see FIG. 10). The image data of this size iscontracted with the aspect ratio thereof (namely, the aspect ratio ofthe display screen of the display 11) maintained so that the image datais displayed within the window area 50B having the size of 640×512pixels assigned to the PC-2. The contracted size is thus the true windowarea size. The true origin is used to place the window of that size atthe center of the window area 508 as shown in FIG. 12, and isrepresented in pixel coordinates at the top left corner of the window(the absolute coordinates with respect to the entire display screen).

The window area information generator 41 determines the above-referencedtrue window area sizes and true origins for the PC-1, PC-2, PC-3, andPDA-1, and generates window area information containing the terminalname item, the window area item, and the origin item as shown in FIG.13, and then stores the window area information in the data storage 26as the window area information file 36 (step S24). The window areainformation generation process thus ends. The window area informationfile 36 is tagged with processing date (May 21, 2002, 17:00:32, forexample).

Returning to FIG. 6, the controller 27 in the display device 2 performsprocesses in steps S12-S14 to each of the terminals 1 a, 1 b, 1 c, and 1d to be displayed (step Slid when the window area information generator41 completes the window area information generation process (step S10).More specifically, the controller 27 places a request to send thecaptured image data (step S12). The sent request to send the capturedimage data includes the true window area size assigned to the terminal 1to which the request is sent, and the display color count of the display21 of the display device 2 stored in the display specificationmanagement file 30. For example, the request to send the captured imagedata including the true window area size of 640×512 (see FIG. 13) andthe display color count of 167,777,216 of the display device 2 is sentto the terminal 1 a (PC-1).

Upon receiving the capture start command sent from the display device 2in step S8, the terminal 1 starts capturing the screen thereof. If thescreen capture performed by the terminal 1 is a full-screen capture, theterminal 1 sends, to the display device 2, the captured image data whichhas been subjected to the size conversion process and the colorconversion process in accordance with the true window area size and thedisplay color count contained in the request to send the captured imagedata, in response to the request to send the captured image data sent instep S12. If the screen capture performed by the terminal 1 is apartial-screen capture, an image size of the partial-captured image issent to the display device 2.

The display device 2 receives the reply from the terminal 1 (step S13).If the rely is the captured image data, the display device 2 determinesthat the screen capture performed by the terminal 1 is a full-screencapture (step S14), and the received captured image data is written ontothe captured image data memory 37 (step S18).

If the reply from the terminal 1 received in step S13 is the image size,the controller 27 determines the screen capture performed by theterminal 1 is a partial-screen capture (step S14). The captured imagesize in the display status management file 34 is updated with thereceived image size. The controller 27 regenerates the window areainformation (the true window area size and the true origin) based on thereceived image size (step S15). The controller 27 updates the requiredportion of the window area information file 38, and sends a request tosend recaptured image data responsive to the regenerated true windowarea size and the display color count of the display 21 (step S16). Theterminal 1 receives the request to send the recaptured image data. Theterminal 1 returns, to the display device 2, the captured image datawhich has been subjected to the size conversion process and the colorconversion process in accordance with the true window area size and thedisplay color count of the display 21, contained in the request to sendthe recaptured image data. The display device 2 receives the reply fromthe terminal 1 (step S17), and writes the reply onto the captured imagedata memory 37 (step S18).

The above process is performed for each of the terminals 1 a-1 d. Whenthe captured image data is received from all terminals 1 a-1 d, thecontroller 27 sends an image synthesis command to the image synthesizer42. The display device 2 enters an image synthesis process (step S19).

Upon receiving the image synthesis command, the image synthesizer 42identifies locations of synthesis of the size-converted captured imagedata and the color-converted captured image data stored in the capturedimage data memory 37 in accordance with the true origin of the windowarea information in the window area information file 36, and synthesizesthe captured image data into a single screen image data, therebygenerating the synthesized image data. The synthesized image data isthen output to the image processor 43.

The image processor 43 converts the synthesized data from the imagesynthesizer 42 in display image data having a scanning frequency of thedisplay 21. The display image data is then output to the display 21. Asshown in FIG. 8, a multi-window screen is thus present on the displayscreen 50 in which the captured image data (display screen) of theterminals 1 a, 1 b, 1 c, and 1 d is presented on the true window areas(hereinafter also referred to as window screens) of 51A, 51B, 51C, and51D (step S20).

The operation of the terminal 1 having received the capture startcommand and the display status management file 34 from the displaydevice 2 is discussed below.

FIG. 14 is a flow diagram of the operation of the terminal 1 which hasreceived the capture start command and the display status managementfile 34 from the display device 2. The terminal 1 is here 1 a (PC-1).

The controller 16 in the terminal 1 a receives, through thecommunication unit 17, the capture start command and the display statusmanagement file 34 sent from the display device 2 (step S31). Thecontroller 16 references the capture area management flag in theterminal 1 a in the display status management file 34 (step S32). Sincethe capture area setting flag is “0” (step S33), the full-screen captureis determined to be activated. The controller 16 sends a full-screencapture command to the screen capture processor 19. In response to thefull-screen capture command, the screen capture processor 19 stores thecontent of the video memory 12 (i.e., the content currently displayed onthe display screen of the display 11) in the storage 15 in a bit-mapformat (a full-screen capture process) (step S34).

When the terminal 1 a receives, from the display device 2, the requestto send the captured image data (step S35), the terminal 1 a performsthe size conversion on the captured image data acquired in thefull-screen capture process in step S34 in accordance with the truewindow area size contained in the request to send the captured imagedata while performs the color conversion process on the captured imagedata in accordance with the display color count contained in the requestto send the captured image data (step S36). Since the terminal 1 a(PC-1) has a screen size of 1280×1024 pixels (see FIG. 10), the capturedimage data of this size is converted (in a contraction process) intodata of size as large as the assigned window area size of 640×512 (seeFIG. 13). No color conversion is performed because the display 11 hasthe same color count of 167,777,216 as the display device 2. If thedisplay 11 has a color count larger than that of the display device 2,the color count is down-converted to match that of the display device 2.The captured image data, size converted and color converted in this way,is sent through the communication unit 17 (step S37).

If the capture area setting flag is determined to be “1” in step S33, apartial capture is determined to be activated. A capture areadesignation screen indicating a message saying “designate a capturearea” is presented on the display 11 (step S38), and a partial capturecommand is sent to the screen capture processor 19. When the user, whoreads the message on the capture area designation screen, selects awindow or encloses a desired area using the input section 13, the screencapture processor 19 recognizes the user operation through the userinterface 14. The image data on the video memory 12 corresponding to thedesignated area and the image size is stored in the storage 15 (apartial capture process) (step S39).

Upon receiving the request to send the captured image data from thedisplay device 2 (step S40), the terminal 1 a returns the size of thepartial captured image captured in the partial capture process in stepS39 (step S41). The reply is received by the display device 2 as alreadydiscussed. The display device 2 regenerates the true window area sizebased on the image size in the partial capture process, and sends, tothe terminal 1 a, the request to send the recaptured image datacontaining the regenerated true window area size and the display colorcount of the display 21 of the display device 2. Upon receiving therequest, the terminal 1 a performs the size conversion process on thepartial captured image data stored in the storage 15 in step S39, basedon the true window area size contained in the request to send thecaptured image data, while performing the color conversion process onthe partial captured image data in accordance with the color countcontained in the request to send the captured image data (step S36). Theterminal 1 a then sends the captured image data, which has been sizeconverted and color converted in this way, to the display device 2through the communication unit 17 (step S37).

The above-referenced process is similarly performed on each of theremaining terminals 1 b, 1 c, and 1 d in addition to the terminal 1 a.As a result, the display device 2 receives, from each of the terminals 1a, 1 b, 1 c, and 1 d, the captured image data which has been sizeconverted into the true window area size assigned thereto, and which hasbeen color converted to the display color count of the display 21 of thedisplay device 2.

Each terminal 1 receives the display status management file 34 from thedisplay device 2. The display status management file 34 is used to checkthe status of the capture area setting flag of own terminal 1.Furthermore, the content of the file may be displayed on the display 11by performing a predetermined operation on the input section 13. Thepermitted connection management file 31, the permitted displaymanagement file 32, and the connection status management file 33 mayalso be acquired from the display device 2 as necessary to be presentedon the display 11. In this way, the user may learn what terminals aredisplayed other than own terminal operated by the user himself and therange of authority granted to own terminal 1.

Expansion Display Function

An expansion display function for expanding any one of a plurality ofcurrently presented windows is discussed below. The expansion displayfunction is performed by updating the priority order in theabove-referenced arrangement.

FIG. 15 diagrammatically illustrates the display system to explain theexpansion display function. The screen of the terminal 1 a is expandedby heightening the priority order of the terminal 1 a.

The screen of the terminal 1 a is designated by operating the remotecontroller. If the terminal 1 a is assigned any key in the remotecontroller, the user designates the terminal 1 a by pressing that key.If no particular key is assigned, the user may operate the remotecontroller to select the terminal 1 a on a menu screen on the display21, or may click the screen of the terminal 1 a with a pointer on thedisplay screen using the remote controller.

Designation information input in this way, i.e., an expansion displayrequest containing the Identification information of the terminal 1 a,is sent to the controller 27 through the user interface 24. Thecontroller 27 identifies the terminal 1 a based on the identificationinformation contained in the expansion display request. The controller27 sets the priority of the terminal 1 a (PC-1) in the display statusmanagement file 34 to be “highest”, and sends a window area splitrequest to the window area information generator 41.

The window area information generator 41 generates the window areainformation file 36 as already described. The tentative window areasetting table, which is referenced in the generation of the window areainformation file 36, is organized as illustrated in FIG. 16A. FIG. 16Bshows tentative window areas based on the tentative window area settingtable. The priority order illustrated in FIG. 16B is determined based onthe priority order in the display status management file 34. Thepriority order of the terminal 1 a is higher than those of the remainingterminals 1 b, 1 c, and 1 d. The terminal 1 a is thus assigned thetentative window area 52A (the window area for the first terminal)having the first priority in FIG. 16B. The assignment of the windowareas to the remaining terminals 1 b, 1 c, and 1 d having no priorityorder set therefor is arbitrary. For example, the terminals 1 b, 1 c,and 1 d are now assigned tentative window areas 52B, 52C, and 52D,respectively. As already described, the true window areas to be assignedto the terminals 1 b, 1 c, and 1 d are determined.

FIG. 17 illustrates the true window areas. The terminals 1 a, 1 b, 1 c,and 1 d are assigned the true window areas 53A, 53B, 53C, and 53D,respectively. The window area information file 36 is then organized asillustrated in FIG. 18.

As already described, the captured image data (display screen) of theterminals 1 b, 1 c, and 1 d appears on the true window areas 53A, 53B,53C, and 53D, respectively. The designated window 51A is shown in anexpanded state thereof on the multi-window screen 50.

Function to Switch to Single-Window Screen

One of the plurality of multi windows is shown on a single-window screenas shown in FIG. 19. The user may operate the remote controller to enteran operational input for the function to switch to a single-windowscreen. More specifically, the controller 27 is notified of the inputinformation, namely, the single-window display request containing theidentification information of the terminal 1 c corresponding to thewindow 51C to be displayed on the single-window screen. The controller27 identifies the terminal 1 c based on the identification informationcontained in the single-window display request. The controller 27 setsthe priority order item of the terminal 1 c (PC-3) in the display statusmanagement file 34 to be the highest, thereby sending a window areasplit request to the window area information generator 41.

As a result, the multi-window screen is replaced with the single-windowscreen as shown in FIG. 19. The user may return to the multi-windowscreen by performing a predetermined operation on the remote controllerto set the priority order to “none”.

The switching to the single-window screen allows the user to recognizedetails, which are not visible in the contracted scale on the window.The ease of use is assured because the predetermined operation quicklyreturns the screen to the multi-window screen.

Insertion Function

As shown in FIG. 20, a new window may be inserted into a currentlypresented multi-window screen. Such an insertion corresponds to adisplay request subsequent to the display of a multi-window screen. Thescreen insertion is thus performed in the same process as in the displayrequest process.

Erase Function

As shown in FIG. 21, one of the plurality of currently presented windowsmay be erased. The user operates the remote controller to enter anoperational input for the erase function. More specifically, the inputinformation, namely, an erase request containing the identificationinformation of the terminal 1 d corresponding to the window 51D to beerased is sent to the controller 27 through the user interface 24. Thecontroller 27 identifies the terminal 1 d based on the identificationinformation contained in the erase request. The controller 27 deletesthe terminal information of the terminal 1 d from the display statusmanagement file 34, and sends a window area split request to the windowarea information generator 41. The subsequent process remains identicalto the one already discussed. The window 500 designated for erasure iserased as shown in FIG. 21. The multi-window screen is reorganized sothat a plurality of windows corresponding to the number of terminalssubsequent to the erasure are presented. Alternatively, the window 51Ddesignated for erasure may be merely erased.

The first preferred embodiment of the present invention provides thedisplay system 100 having the multi-window screen presentation functionin which the screens of the plurality of terminals 1 connected to thenetwork are presented on the plurality of windows on the display screenof the display device 2. The processes required to perform themulti-window screen presentation function, i.e., the size conversionprocess and the color conversion process, are performed on the terminal1. This arrangement dramatically reduces the workload on the displaydevice 2, in comparison with the case in which the display device 2performs the same processes on the captured image data sent from theterminal 1. Since the size conversion process and the color conversionprocess are performed by the terminal 1, the effect of an increase inthe number of terminals 1 on processes to be performed by the displaydevice 2 is minimized in the display system 100.

The workload on the network 3 is also reduced because the terminal 1performs the size conversion process on the captured image data beforethe transmission of the captured image data to the display device 2 overthe network 3.

The ease of use of the system 100 is assured because the expansiondisplay function, the switching function to the single-window screen,the insertion function, and the erase function are available in additionto the multi-window screen presentation function.

The captured image data acquired in the terminal 1 is presented on thecorresponding window on the display screen of the display device withthe aspect ratio thereof maintained. The display screen of the displaydevice is thus free from discordance.

Rather than updating the display screen each time the conferenceparticipant (the user) places the display request through the inputsection 23, the display device 2 itself updates the display screenthereof every three seconds, for example. In this case, the controller27 in the display device 2 controls timings, thereby performingsubsequent processes as described above at regular intervals. Theconstantly updated display screen of each terminal 1 is viewed on thedisplay of the display device 2.

Difference Capture Function

The full-screen capture method and the partial-screen capture methodhave been described. The display system 100 also provides a method inwhich a change in the display screen on the terminal 1 is detected andimage data obtained by capturing the change only is sent.

The user operates the remote controller of the input section 23 to enteran operational input to perform a difference capture on each of theterminals 1 a-1 d. More specifically, the difference capture managementflag in the display status management file 34 is set to be “1”.

For a first cycle of process immediately subsequent to the setting ofthe difference capture management flag to “1”, steps S1-S20 areperformed as described above to present the multi-window screen.

After the completion of the multi-window presentation, the controller 27in the display device 2 sends a difference capture start commandtogether with the display status management file 34 to the terminal 1with the difference capture management flag thereof set to “1” throughthe communication unit 28 and the network 3.

The operation of the terminal 1 having received the difference capturestart command and the display status management file 34 from the displaydevice 2 will be discussed below.

FIG. 22 is a flow diagram illustrating the operation of terminal whichhas received the difference capture start command and the display statusmanagement file 34 from the display device 2. Here, the terminal ofinterest is the terminal 1 a (PC-1).

The data management processor 18 in the terminal 1 a receives, throughthe communication unit 17, the difference capture start command and thedisplay status management file 34 sent by the display device 2 (stepS41). The terminal 1 a thus performs a subsequent full-screen captureprocess. More specifically, the screen capture processor 19 stores thecontent of the video memory 12 in the storage 15 (step S42). The imagedata obtained here is referred to as pre-full-screen data.

The screen comparison processor 20 references the difference capturemanagement flag in the display status management file 34, received bythe data management processor 18, at any regular intervals set (onceevery 0.5 second, for example) (step S43). If the difference capturemanagement flag is “1” (step S44), a subsequent difference captureoperation is performed.

To quit the difference capture operation, the user simply sets thedifference capture management flag to “0” In the same way as the displaystatus management file 34 is modified. The screen comparison processor20 determines whether or not the capture operation is suspended, byreferencing the difference capture management flag.

After performing the full-screen capture process, the screen captureprocessor 19 stores the content of the video memory 12 in the storage 15(step S45). The image data obtained here is referred to aspost-full-screen data. The screen comparison processor 20 in thecontroller 16 compares the pre-full-screen data with thepost-full-screen data (step S46).

A screen comparison process is discussed in which a mouse pointer ismoved (see FIG. 23). The mouse pointer changes the position thereof frompre-full-screen data 60 and post-full-screen data 61. As shown in FIG.24, two areas 70 and 71 are recognized as being changed. The screencomparison processor 20 detects a change. If it is determined that therehas been a change in the screen (step S47), the screen comparisonprocessor 20 acquires image data of an area that has undergone a change,a size of the image data (the number of pixels in vertical andhorizontal directions), and coordinates of an origin of the image data(absolute coordinates within the area defined by the captured image sizein the display status management file 34).

In this case, the screen comparison processor 20 acquires, with respectto the area 70, captured image data of the area 70 (hereinafter referredto difference captured image data 1), a difference captured image size1, and a difference capture origin 1, and with respect to the area 71,captured image data of the area 71 (hereinafter referred to differencecaptured image data 2), a difference captured image size 2, and adifference capture origin 2. The screen comparison processor 20 thenstores these pieces of information in the storage 15 while notifying thedata management processor 18 that the difference captured image data hasbeen acquired (step S48). The difference captured image size 1, thedifference captured image size 2, the difference capture origin 1, andthe difference capture origin 2 are written on the respective portionsthereof in the display status management file 34.

The data management processor 18 sends these pieces of data to thedisplay device 2 through the communication unit 17 and the network 3(step S49). The difference capture function is different from thefull-screen capture and the partial-screen capture. More specifically,the difference capture function is performed not in response to therequest to send received from the display device 2 but in response tothe change in the screen detected by the screen comparison processor 20.The transmission operation of the captured image data is then performed.

The post-full-screen data is set to be pre-full-screen data for asubsequent screen comparison process (step S50).

The display device 2 then receives the difference captured image data 1,the difference captured image data 2, and the display status managementfile 34 from the terminal 1 a. The operation of the display device 2subsequent to the reception of these pieces of data will now bediscussed.

Upon receiving the difference captured image data 1, the differencecaptured image data 2, and the display status management file 34 fromthe terminal 1 a at the communication unit 28, the controller 27transfers an image synthesis command to the image synthesizer 42. Thedisplay device 2 enters the image synthesis process in the differencecapture function.

Upon receiving the image synthesis command, the image synthesizer 42rewrites portions of the captured image data stored in the capturedimage data memory 37 corresponding to the difference captured image data1 and the difference captured image data 2, based on the window areainformation file 36 corresponding to the image displayed on the display21 and the display status management file 34 (the difference capturedimage size 1, the difference captured image size 2, the differencecapture origin 1, and the difference capture origin 2) received from theterminal 1 a. The process subsequent to this operation are identical tothose in steps S19 and S20.

The amount of image data transmitted over the network 3 is smaller inthe difference capture process than in the full-screen capture (or thepartial-screen capture) process. The workload on the network 3 is thusreduced. The user constantly monitors an updated image of the terminal1.

The above description of the difference capture process is based on theassumption that a dedicated program installed on the terminal 1 is usedto capture the screen. If a driver for directly detecting a differencein the content of the video memory is available on the operating system(OS), such a driver may be used.

The image data may be exchanged in a compression standard format (suchas JPEG) between the terminal 1 and the display device 2 no reduce theworkload on the network 3.

In the above description, a variety of requests such as the displayrequest and the single-window screen display request is performed byoperating the input section 23 on the display device 2. In other words,the display device 2 has the initiative in the organization of thescreen.

Alternatively, each terminal 1 may have the initiative. The conferenceparticipant (the user) enters a desired operational input by operatingthe input section 13 on own terminal 1. The input information istransferred to the controller 16 through the user interface 14. Thecontroller 16 in turn sends the request, including the terminal name andthe IP address of own terminal 1, responsive to the input information tothe display device 2 via the communication unit 17 and the network 3.The request is received by the communication unit 28 in the displaydevice 2 through the network 3. The request is then transferred to thecontroller 27. The operation subsequent thereto remains the same asthose already discussed. In this way, a variety of requests may beplaced using the terminal 1.

The terminal 1 may designate the capture area by containing information,which designates the full-screen capture or the partial-screen capture,in the variety of requests.

The priority order may be designated by the user as necessary asdescribed above. Alternatively, a plurality of terminals 1 which arescheduled to be connected to the display device 2 may be assignedbeforehand priority order. If the terminals 1 are assigned beforehandthe priority order, the priority assigned to each terminal 1 isautomatically set in the priority item of the display status managementfile 34 when the terminal information is registered in the displaystatus management file 34.

In the above description, the tentative window area is determined basedon the tentative window area setting file 35. Alternatively, thetentative window area may be determined through calculation each time.

In the partial capture, the terminal 1 returns the image size to thedisplay device 2. The display device 2 regenerates the true window areain response to the received image size. The regeneration of the truewindow area may be performed by the terminal 1.

Second Embodiment

FIG. 1 illustrates a network of a display system 100 including a networkinteractive display device 2 in accordance with preferred embodiments ofthe present invention.

The display system 100 includes a plurality of terminals (only fourterminals 1 a, 1 b, 1 c, and 1 d are shown in FIG. 1), and the networkinteractive display device 2 (a projector here) having a multi-windowscreen presentation function as one of major functions of the presentinvention. The plurality of terminals 1 are respectively connected tothe network interactive display device 2 through a network 3 in atwo-way communication based on the TCP/IP protocol. A unique name isprovided beforehand to each terminal 1 (hereinafter referred to as aterminal name). The network 3 may be any of a LAN (Local Area Network),a radio LAN, and a near-field communication radio LAN such as Bluetooth(Tradename of Bluetooth SIG Inc., U.S.A.).

The display system 100 allows screens presented on the plurality ofterminals 1 to be concurrently presented on a multi-window displayscreen of the network interactive display device 2. Such a system 100 isuseful in a conference or a presentation. The terminal 1 and the networkinteractive display device 2 will now be discussed in detail.

FIG. 26 is a block diagram illustrating the structure of the terminal 1in accordance with a second preferred embodiment of the presentinvention.

The terminal 1 may be a personal computer or a PDA (Personal DigitalAssistant). The terminal 1 includes a display 11 for presenting adiversity of information such as materials for presentation, a videomemory 12 for storing the content to be presented on the display 11, aninput section 13 including a tablet, a mouse, or a keyboard, a userinterface 14 for detecting an operational input from the input section13 and outputting the operational input to an arithmetic unit (CPU) 16,a storage 15 for storing application software programs (such as acontrol program) for performing the processes of the present invention,the arithmetic unit 16, and a communication unit 17.

The control program stored in the storage 15 is used to perform aterminal control function to achieve a multi-window screen presentationfunction on the network interactive display device 2, a screen capturefunction to capture a whole or a part of the screen of the display 11,an image conversion function to convert captured image data acquired bythe screen capture function into data in a format of a display 21 of thenetwork interactive display device 2, and a function to detect a changeon the screen of the display 11. The application software programs andthe CPU constitute a data management processor 18, a screen captureprocessor 19, and a screen comparison processor 20.

The data management processor 18 receives a variety of requests,including a connection request, a display request, an expansion displayrequest, a request to switch to a single-window full screen, and anerase request through the user interface 14 or the communication unit17, and performs processes responsive to each request. The datamanagement processor 18 sends the captured image data, acquired by theimage capture processor 19, to the display device 2 through thecommunication unit 17.

The communication unit 17 carries out a two-way communication with thenetwork interactive display device 2. The communication protocol usedhere is the TCP/IP. The communication unit 17 has a protocol processingfunction for ARPICMP, IP, TCP, UDP, etc. required for the TCP/IPconnection. This protocol processing function is carried out under thecontrol of an OS.

FIG. 27 is a block diagram illustrating the structure of the networkinteractive display device 2 of the second preferred embodiment.

The display device 2 includes the display 21, a display control unit 22which has a multi-window screen presentation function, an expansiondisplay function, a function to switch to the single-window full screen,an insertion function, and an erase function, and controls the displayscreen to be presented on the display 21, an input section 23 includinga remote controller, a mouse, or a keyboard, a user interface 24 fordetecting an operational input from the input section 23 and foroutputting the operational input to an arithmetic unit 27 to bediscussed later, a program storage 25 for storing the display controlprogram to provide the multi-window screen presentation function of thepresent invention, a data storage 26 for storing a variety of files anddata required to carry out a control program, the arithmetic unit (CPU)27 for generally controlling the display device 2, and a communicationunit 28 for performing a two-way communication with each terminal 1.

The communication unit 28 carries out a two-way communication with theterminal 1. The communication protocol used here is the TCP/IP. Thecommunication unit 23 has a protocol processing function for ARPICMP,IP, TCP, UDP, etc. required for the TCP/IP connection.

The display device 2 may be a plasma display, or a liquid-crystaldisplay instead of the projector shown in FIG. 1. The display 21 isdifferent depending on the type of the display device 2. Specifically,the display 21 is one of a liquid-crystal light valve, an LCoS lightvalve, or a DMD (Digital Micromirror Device) (Trademark of TexasInstruments) in the projector. The display screen of the display 21becomes a projecting screen. The display 21 is a plasma display panel inthe plasma display device, a liquid-crystal panel in the liquid-crystaldisplay device, or an organic EL (Electroluminescent) panel in theorganic EL display device.

Referring to FIG. 28, a variety of files stored in the data storage 26are discussed.

The data storage 26 stores a permitted connection management file 31, apermitted display management file 32, a connection status managementfile 33, a display status management file 34, a tentative window areasetting file 35, and a window area information file 36. The data storage26 further includes a captured image data memory 37 for storing thecaptured image data sent from each terminal 1.

The permitted connection management file 31 registers a terminal name ofa terminal 1 which is permitted for connection. The permitted displaymanagement file 32 registers a terminal name of a terminal 1 which ispermitted for screen display.

The connection status management file 33 registers a terminal name of aterminal 1 which is currently connected to the display device 2.

The display status management file 34 manages a display status of thecurrent display 21. The display status management file 34 manages, in atable form, terminal information relating to the terminal 1 that is asource of the captured image data currently presented on the displayscreen of the display 21. The display status management file 34 isupdated each time the display screen of the display 21 is modified. Forexample, if the display screen is switched from a four-window screen toa three-window screen, terminal information of the terminal 1corresponding to an erased window is deleted. If the display screen isswitched from a four-window screen to a five-window screen, terminalinformation of the terminal 1 corresponding to an added window is newlyregistered.

FIG. 5 illustrates items of terminal information managed in the displaystatus management file 34.

The display status management file 34 contains, as items thereof, a“terminal name”, an “IP address”, a “screen size”, “color countinformation”, “priority”, a “capture area management flag”, a“difference capture management flag”, a “captured image size”, a“difference captured image size 1”, a “difference captured image size2”, a “difference capture origin 1”, and a “difference capture origin2”.

The “terminal name” is a name provided beforehand to the terminal 1. The“screen size” is the number of pixels in the vertical and horizontaldirections of the display screen of the display 11. For example, an SXGAterminal has 1280×1024 pixels, and an XGA terminal has 1024×768 pixels.The “color count information” represents the number of display colors ofthe display 11, and may be 256 colors, or 167,777,216 colors, forexample. The terminal name, the IP address, the screen size, and thecolor count information are the items that must be stored in the displaystatus management file 34 during a registration. Other items are set(updated) by the user as necessary.

The “priority” determines the size of the display size of each windowassigned to the terminal 1 that is identified by the terminal name. Thepriority order takes “highest”, “high” or “none”. As will be discussedin detail, the display size having a high priority order becomes large.The “capture area management flag” manages the capture as to whether thescreen of the terminal 1 identified by the terminal name is captured ina full-screen capture mode or a partial-screen capture mode. The capturearea management flag is “0” in the full-screen capture mode, which is astandard mode, or “1” in the partial-screen capture mode.

The “difference capture management flag” manages the capture of whetherthe screen of the terminal 1 identified by the terminal name is capturedin a normal capture mode or a change capture mode (hereinafterreferenced to as a difference capture mode) for only a change on thedisplay screen. The difference capture management file is “0” in thenormal capture mode, or “1” In the difference capture mode.

The “captured image size” is the size of the captured image data (thenumber of pixels in the vertical and horizontal directions) when thecapture area management flag is “1”, i.e., in the partial-screen capturemode.

The “difference captured image size 1” and the “difference capturedimage size 2” represent the sizes of two different areas acquired in thedifference capture when the difference capture management flag is “1”.The “difference capture origin 1”, and the “difference capture origin 2”are the origins of the two different areas acquired in the differencecapture, and are the absolute coordinates within an area defined by thecaptured image size.

The tentative window area setting file 35 is a file in which informationidentifying a tentative window area assigned to the terminal 1 is setbeforehand. The tentative window area setting file 35 contains aplurality of tables, each table prepared for the terminals. The tablehas a structure as shown in FIG. 11A and FIG. 16A. The table will bediscussed later. The window area information file 36 will also bediscussed later.

When a predetermined operation is performed on the input section 23, thedisplay 21 displays the contents of the files 31, 32, 33, 34, 35, and36. The user thus checks and modifies the data on the display screen atwill.

Returning to FIG. 28, the display control unit 22 includes a window areainformation generator 41, an image synthesizer 42, and an imageprocessor 43. The arithmetic unit 27 receives a variety of requests suchas the connection request, the display request, the insertion displayrequest, the erase request, etc., received through the user interface 24or the communication unit 28. In response to these requests, under thecontrol of the arithmetic unit 27, the processors 41, 42, and 43respectively perform required processes while accessing necessary filesin the data storage 26. The arithmetic unit 27 thus controls the display21. A display control program, stored in the program storage 25, forproviding a multi-window screen presentation function and the arithmeticunit (CPU) 27 constitute the display control unit 22.

From the display status management file 34, the window area informationgenerator 41 learns the number of terminals 1 to be presented, and thepriority order and the screen size of each terminal 1. The window areainformation generator 41 splits the display screen size of the display21 in accordance with the number of terminals 1 to be presented, and thepriority order and the screen size of each terminal 1. The window areainformation generator 41 generates the window area informationcontaining display size (hereinafter referred to as a window area size)of the window on the display 21 assigned to each terminal 1 to bedisplayed, and information identifying the display position of thewindow (the absolute coordinates at the top left corner of the windowwith respect to the display screen, hereinafter also referred to as anorigin). The information is stored in the data storage 26 as the windowarea information file 36.

The image synthesizer 42 performs various processes including acontraction process, an expansion process, and a color conversionprocess on the captured image data stored in the captured image datamemory 37, and then synthesizes the captured image data in accordancewith the window area information file 36 generated in the window areainformation generator 41.

The image processor 43 performs a scanning frequency conversion processon a variety of pieces of image data such as the synthesized image datagenerated in the image synthesizer 42, and the display status managementfile 34 of the data storage 26 which is referenced using an OSD(on-screen display) function, thereby generating display image data andoutputting the display image data to the display 21. The image processor43 includes a scan converter, for example.

The operation of the second preferred embodiment of the presentinvention will now be discussed. FIG. 29 is a flow diagram illustratingthe operation of the second preferred embodiment.

A predetermined operational input is entered in the input section 23 inthe display device 2 in a preliminary step for multi-window screenpresentation. Upon detecting the operational input through the userinterface 24, the arithmetic unit 27 broadcasts a request to return aterminal name and an IP address together with the IP address of thedisplay device 2 through the communication unit 28 and the network 3.When each terminal 1 receives the broadcast request to return theterminal name and the IP address, the terminal 1 returns own terminalname and IP address to the display device 2.

The display device 2 receives a reply (the terminal name and the IPaddress) from each terminal 1 through the communication unit 28, anddetermines whether each terminal is a connection permitted terminal.Specifically, the display device 2 determines whether the returnedterminal name agrees with a terminal name registered in the permittedconnection management file 31. If it is determined that the returnedterminal name agrees with the registered terminal name, the displaydevice 2 handles the terminal 1 as a connection permitted terminal.

The terminal names and the IP addresses of the terminals 1 determined asconnection permitted terminals are successively registered in theconnection status management file 33. The connection status managementfile 33 allows the display device 2 to learn how many terminals 1 arecurrently connected. Since the determination of whether the connectionis permitted or not is based on the terminal name, the system works evenif the IP address, provided to the terminal 1 using the DHCP, becomesdifferent each time connection is made.

The display device 2 waits on standby for any request after the abovepreliminary step 1 s complete. The display screens of the four terminals1 a-1 d, out of the terminals 1 operated by conference participants, arepresented on a multi-window display screen 50 of the display device 2.As for the resolutions thereof, the terminal 1 a has an SXGA resolution(1280×1024 pixels), the terminal 1 b has an SVGA resolution (800×600pixels), the terminal 1 c has an XGA resolution (1024×768 pixels), andthe terminal 1 d has a resolution of 480×640 pixels.

Multi-Window Screen Presentation Function

FIG. 8 illustrates the configuration of the display system in whichdisplay screens of four terminals 1 a-1 d are presented on the displayscreen of the display device 2.

The user operates the remote controller in the input section 23 to enteran operational input to display the screens of the terminals 1 a-1 d.Through the user interface 24, the arithmetic unit 27 is notified of theinput information, namely, a request to display the screens of theterminals 1 a-1 d together with identification information of theterminals 1 a-1 d (step S1). When the request is placed, the priorityorder, the partial capture, and the difference capture may bedesignated. Here, no particular designation is performed.

Upon receiving the display request, the arithmetic unit 27 in thedisplay device 2 performs processes in steps S3-S9 for each of theterminals 1 a-1 d to be displayed (step S2). The arithmetic unit 27references the permitted connection management file 31 and the permitteddisplay management file 32 according to the terminal name indicated bythe identification information contained in the display request, therebydetermining whether or not each terminal 1 is permitted for connectionand whether or not each terminal 1 is permitted for display (step S3).If it is determined that each terminal 1 is permitted for bothconnection and display (step S4), the arithmetic unit 27 requests,through the communication unit 28, each terminal 1 to send the terminalinformation (the terminal name, the IP address, the screen size, and thecolor count information) (step S5). The arithmetic unit 27 receives theterminal information which has been sent in response to the request(step S6), and registers the terminal information in the display statusmanagement file 34 (step S7). If the priority order, the partialcapture, and the difference capture are designated during the placementof the display request, the priority order, the capture area managementflag, and the difference capture management flag are also registered inthe registration in step S7.

The arithmetic unit 27 sends the display status management file 34 and ascreen capture start command to the terminal 1 which is permitted forconnection and display (step S8). When any terminal 1 is not permittedfor connection and display, the arithmetic unit 27 sends a notificationto that effect to the terminal 1 (step S9).

If all four terminals 1 a-1 d are permitted for connection and display,the terminal information from the terminals 1 a-1 d is registered in thedisplay status management file 34 in the processes in steps S2-S9. Atthe same time, the screen capture start command is sent together withthe display status management file 34 to each of the terminals 1 a-1 dthrough the communication unit 28.

Subsequent to the above processes, the arithmetic unit 27 notifies thewindow area information generator 41 in the display control unit 22 of awindow area split request. The display device 2 enters a window areainformation generation process (step S10). The operation of the terminal1 having received the screen capture start command will be discussedlater. Discussed first is the window area information generation processperformed by the window area information generator 41 in response to thewindow area split request.

FIG. 9 is a flow diagram illustrating the flow of a window areainformation generation process. The operation of the window areainformation generator 41 is specifically discussed on the assumptionthat the display status management file 34 is constructed as shown inFIG. 10. As shown in FIG. 10, terminal names PC-1, PC-2, PC-3, and PDA-1correspond to the terminals 1 a, 1 b, 1 c, and 1 d, respectively.

Upon receiving the window area split request from the arithmetic unit27, the window area information generator 41 learns the number of theterminals 1 to be displayed (here, four terminals 1) referencing thedisplay status management file 34. The user interface 14 also acquiresthe priority order of each of the terminals 1 a-1 d (step S21). Thewindow area information generator 41 references the tentative windowarea setting file 35 according to the number of terminals 1 and thepriority order of each of the terminals 1 a-1 d, and acquires atentative size and a tentative origin of each tentative window areaassigned to each of the terminals 1 a, 1 b, 1 c, and 1 d (step S22). Aswill be clarified later, the adjective “tentative” is used because thearea window here assigned to the terminal 1 is updated in a later stepto size convert the captured image data.

As shown in FIG. 10, the four terminals 1 are to be displayed here andno priority order is set to all of the four terminals 1. A tentativearea setting table in the tentative window area setting file 35 isorganized as shown in FIG. 11A. Here, the display screen of the display21 has a resolution of 1280×1024 (SXGA), and the tentative area settingtable shown in FIG. 11A is organized based on this display screen. FIG.11B shows the tentative window area based on the table window areasetting table shown in FIG. 11A.

The priority order shown in FIG. 11A is determined based on the“priority” item in the display status management file 34, and theterminals 1 are first, second, third, and fourth in from the high orderto the low order. The terminals 1 are assigned the “tentative size” andthe “tentative origin” for the tentative window area in the lower table.All the terminals 1 a-1 d have “none” In the priority order row with nopriority order set therefor (see FIG. 10). If no priority order is set,the order of assignment may be a predetermined one, or may be the orderof registration to the display status management file 34. In the secondpreferred embodiment, the terminals 1 a, 1 b, 1 c, and 1 d (hereinafterreferred to as the terminal names PC-1, PC-2, PC-3, and PDA-1 asappropriate) are assigned tentative window areas 50A, 50B, 50C, and 50Din that order.

The window area information generator 41 further acquires the screensizes of the PC-1, PC-2, PC-3, and PDA-1 from the display statusmanagement file 34 (see FIG. 10), and determines the sizes and originsof true windows respectively assigned thereto based on the acquiredscreen sizes (step S23).

FIG. 12 illustrates the true window area size and the true origin. Asshown, 51A, SIB, 51C, and 51D represent the true window areas assignedto the PC-1, PC-2, PC-3, and PDA-1, respectively. The captured imagedata to be displayed on the tentative window area is size converted withthe aspect ratio thereof maintained. The true window areas are displayareas of the tentative window areas 50A, 50B, 50C, and 50D in which theconverted images are respectively displayed with the centers thereofaligned to be centered on the respective tentative window areas. ThePC-2 is now specifically discussed in connection with the true windowarea size to determine the true window area. The screen size of the PC-2is 1024×768 pixels (see FIG. 10). The image data of this size iscontracted with the aspect ratio thereof (namely, the aspect ratio ofthe display screen of the display 11) maintained so that the image datais displayed within the window area 50B having the size of 640×512pixels assigned to the PC-2. The contracted size is thus the true windowarea size. The true origin is used to place the window of that size atthe center of the window area 50B as shown in FIG. 12, and isrepresented in pixel coordinates at the top left corner of the window(the absolute coordinates with respect to the entire display screen).

The window area information generator 41 determines the above-referencedtrue window area sizes and true origins for the PC-1, PC-2, PC-3, andPDA-1, and generates window area information containing the terminalname item, the window area item, and the origin item as shown in FIG.13, and then stores the window area information in the data storage 26as the window area information file 36 (step S24). The window areainformation generation process thus ends. The window area informationfile 36 is tagged with processing date (May 21, 2002, 17:00:32, forexample).

Returning to FIG. 29, the arithmetic unit 27 in the display device 2performs processes in steps S12-S14 to each of the terminals 1 a, 1 b, 1c, and 1 d to be displayed (step S11) when the window area informationgenerator 41 completes the window area information generation process(step S10). More specifically, the arithmetic unit 27 places a requestto send the captured image data (step S12). Each of the terminals 1 a, 1b, 1 c, and 1 d has already started the image capture process afterreceiving the capture start command transmitted from the display device2 in step S8. The captured image data acquired in the screen captureprocess and the image size (in the partial capture process) are sent tothe display device 2 in response to the request to send the capturedimage data in step S11.

The display device 2 receives the reply from the terminal 1 (step S13).If the rely is the captured image data, the display device 2 determinesthat the screen capture performed by the terminal 1 is the full-screencapture (step S14), and the received captured image data is written ontothe captured image data memory 37 (step S16).

If the reply from the terminal 1 received in step S13 contains thecaptured image data and the image size, the arithmetic unit 27determines the screen capture performed by the terminal 1 is apartial-screen capture (step S14). The captured image size in thedisplay status management file 34 is updated with the received imagesize. The arithmetic unit 27 regenerates the window area information(the true window area size and the true origin) based on the receivedimage size (step S15). The arithmetic unit 27 writes the receivedcaptured image data together with the image size on the captured imagedata memory 37 (step S16).

The above process is performed for each of the terminals 1 a-1 d. Whenthe captured image data is received from the terminals 1 a-1 d, thearithmetic unit 27 sends an image synthesis command to the imagesynthesizer 42. The display device 2 enters the image synthesis process(step S17).

Upon receiving the image synthesis command, the image synthesizer 42performs the size conversion process and the color conversion process onthe captured image data stored in the captured image data memory 37based on the window area information in the window area information file36 and the color count information in the display status management file34. The image synthesizer 42 then the captured image data into a singlescreen image data, thereby generating the synthesized image data. Thesynthesized image data is then output to the image processor 43.

The image processor 43 converts the synthesized data from the imagesynthesizer 42 in display image data having a scanning frequency of thedisplay 21. The display image data is then output to the display 21. Asshown in FIG. 8, a multi-window display is thus present on the displayscreen 50 in which the captured image data (display screen) of theterminals 1 a, 1 b, 1 c, and 1 d is presented on the true window areas(hereinafter also referred to as window screens) of 51A, 51B, 51C, and51D (step S18).

The operation of the terminal 1 having received the capture startcommand and the display status management file 34 from the displaydevice 2 is discussed below.

FIG. 30 is a flow diagram of an operation of the terminal 1 which hasreceived the capture start command and the display status managementfile 34 from the display device 2. The terminal 1 is here 1 a (PC-1).

The data management processor 18 in the terminal 1 a receives, throughthe communication unit 17, the capture start command and the displaystatus management file 34 sent from the display device 2 (step S31). Thedata management processor 18 references the capture area setting flag inthe terminal 1 a in the display status management file 34 (step S32).Since the capture area setting flag is “0” (step S33), the full-screencapture is determined to be activated. The data management processor 18sends a full-screen capture command to the screen capture processor 19.In response to the full-screen capture command, the screen captureprocessor 19 stores the content of the video memory 12 (i.e., thecontent currently displayed on the display screen of the display 11) inthe storage 15 in a bit-map format (the full-screen capture process)(step S34). The image capture processor 19 notifies the data managementprocessor 18 of the completion of the screen capture process.

If the capture area setting flag is “1” in step S33, a partial captureis determined to be activated. A capture area designation screenindicating a message saying “designate a capture area” is presented onthe display 11 (step S35), and a partial capture command is sent to thescreen capture processor 19. When the user, who reads the message on thecapture area designation screen, selects a window or encloses a desiredarea using the input section 13, the screen capture processor 19recognizes the user operation through the user interface 14. The imagedata on the video memory 12 corresponding to the designated area and theimage size is stored in the storage 15 (the partial capture process)(step S36). The image capture processor 19 notifies the data managementprocessor 18 of the completion of the screen capture process.

Upon receiving the capture start command, the terminal 1 a performs thescreen capture process. Meanwhile, the display device 2 performs thewindow area information generation process as already described. Theterminal 1 a sends the captured image data and the image size (in thepartial capture) to the display device 2 through the communication unit17 (step S38) after the screen capture processor 19 completes the screencapture process and the request to send the captured image data isreceived from the display device 2 (step S37).

The above-referenced process is similarly performed on each of theremaining terminals 1 b, 1 c, and 1 d in addition to the terminal 1 a.As a result, the display device 2 receives, from each of the terminals 1a, 1 b, 1 c, and 1 d, the captured image data and the image size (in thepartial capture process).

Each terminal 1 receives the display status management file 34 from thedisplay device 2. The display status management file 34 is used to checkthe status of the capture area setting flag of own terminal 1.Furthermore, the content of the file may be displayed on the display 11by performing a predetermined operation on the input section 13. Thepermitted connection management file 31, the permitted displaymanagement file 32, and the connection status management file 33 mayalso be acquired from the display device 2 as necessary to be presentedon the display 11. In this way, the user may learn what terminals aredisplayed other than own terminal operated by the user himself and therange of authority granted to own terminal 1.

Expansion Display Function

The expansion display function for expanding any one of a plurality ofcurrently presented windows is discussed below. The expansion displayfunction is performed by updating the priority order in theabove-referenced arrangement.

FIG. 15 diagrammatically illustrates the display system to explain theexpansion display function. The screen of the terminal 1 a is expandedby heightening the priority order of the terminal 1 a.

The screen of the terminal 1 a is designated by operating the remotecontroller. If the terminal 1 a is assigned any key in the remotecontroller, the user designates the terminal 1 a by pressing that key.If no particular key is assigned, the user may operate the remotecontroller to select the terminal 1 a on a menu screen on the display21, or may click the screen of the terminal 1 a with a pointer on thedisplay screen using the remote controller.

Designation information input in this way, i.e., the expansion displayrequest containing the identification information of the terminal 1 a.Is sent to the arithmetic unit 27 through the user interface 24. Thearithmetic unit 27 identifies the terminal 1 a based on theIdentification information contained in the expansion display request.The arithmetic unit 27 sets the priority order of the terminal 1 a(PC-1) in the display status management file 34 to be “highest”, andsends a window area split request to the window area informationgenerator 41.

The window area information generator 41 generates the window areainformation file 36 as already described. The tentative window areasetting table, which is referenced in the generation of the window areainformation file 36, is organized as illustrated in FIG. 16A. FIG. 16Bshows tentative window areas based on the tentative window area settingtable. The priority order illustrated in FIG. 16B is determined based onthe priority order in the display status management file 34. Thepriority order of the terminal 1 a is higher than those of the remainingterminals 1 b, 1 c, and 1 d. The terminal 1 a is thus assigned thetentative window area 62A (the window area for the first terminal)having the first priority in FIG. 16B. The assignment of the windowareas to the remaining terminals 1 b, 1 c, and 1 d having no priorityorder set therefor is arbitrary. For example, the terminals 1 b, 1 c,and 1 d are now assigned the window areas 52B, 52C, and 52D,respectively. As already described, the true window areas to be assignedto the terminals 1 b, 1 c, and 1 d are determined.

FIG. 17 illustrates the true window areas. The terminals 1 a, 1 b, 1 c,and 1 d are assigned the true window areas 53A, 53B, 53-C, and 53D,respectively. The window area information file 36 is then organized asillustrated in FIG. 18.

As already described, the captured image data (display screen) of theterminals 1 b, 1 c, and 1 d appears on the true window areas 53A, 53B,53C, and 53D, respectively. The designated window 53A is shown in theexpanded state on the multi-window screen 50 as shown in FIG. 15.

Function to Switch to Single-Window Screen

One of the plurality of multi windows is shown on a single-window screenas shown in FIG. 19. The user operates the remote controller to enter anoperational input for the function to switch to a single-window screen.More specifically, the arithmetic unit 27 is notified of the inputinformation, namely, the single-window display request containing theidentification information of the terminal 1 c corresponding to thewindow 51C to be displayed on the single-window screen, through the userinterface 24. The arithmetic unit 27 identifies the terminal 1 c basedon the Identification information contained in the single-window displayrequest. The arithmetic unit 27 sets the priority order item of theterminal 1 c (PC-3) in the display status management file 34 to be thehighest, thereby sending a window area split request to the window areainformation generator 41.

As a result, the multi-window screen is replaced with the single-windowscreen as shown in FIG. 19. The user may return to the multi-windowscreen by performing a predetermined operation on the remote controllerto set the priority order to “none”.

The switching to the single-window screen allows the user to recognizedetails, which are not visible in the contracted scale on the window.The ease of use is assured because the predetermined operation quicklyreturns the screen to the multi-window screen.

Insertion Function

As shown in FIG. 20, a new screen may be inserted into a currentlypresented multi-window screen. Such an insertion corresponds to adisplay request subsequent to the display of a multi-window screen. Thescreen insertion is thus performed in the same process as in the displayrequest process.

Erase Function

As shown in FIG. 21, one of the plurality of currently presented windowsmay be erased. The user operates the remote controller to enter anoperational input for the erase function. More specifically, the inputinformation, namely, an erase request containing the identificationinformation of the terminal 1 d corresponding to the window 51D to beerased is sent to the arithmetic unit 27 through the user interface 24.The arithmetic unit 27 identifies the terminal 1 d based on theidentification information contained in the erase request. Thearithmetic unit 27 deletes the terminal information of the terminal 1 dfrom the display status management file 34, and sends a window areasplit request to the window area information generator 41. Thesubsequent process remains identical to the one already discussed. Thewindow 50D designated for erasure is erased as shown in FIG. 21. Themulti-window screen is reorganized so that a plurality of windowscorresponding to the number of terminals subsequent to the erasure arepresented. Alternatively, the window 51D designated for erasure may bemerely erased.

In accordance with the second preferred embodiment, the screensrespectively presented on the plurality of terminals 1 connected to thenetwork 3 are presented on the display of the display device 2 havingthe multi-window screen presentation function.

Since the expansion display function, the switching function to thesingle-window screen, the insertion function, and the erase function areavailable in addition to the multi-window screen presentation function,a sophisticated display device 2 is provided.

The size conversion is performed with the aspect ratio of the capturedimage data maintained when a multi-window screen is presented. Thedisplay device 2 thus presents a display screen free from discordance.

Rather than updating the display screen each time the conferenceparticipant (the user) places the display request through the inputsection 23, the display device 2 itself updates the display screenthereof every three seconds, for example. In this case, the arithmeticunit 27 in the display device 2 controls timings, thereby performingsubsequent processes as described above at regular intervals. Thedisplay screen of each terminal 1 is constantly updated.

Difference Capture Function

The full-screen capture method and the partial-screen capture methodhave been described. The display system 100 also provides a method inwhich a change in the display screen on the terminal 1 is detected andimage data obtained by capturing the change only is sent.

The user operates the remote controller of the input section 23 to enteran operational input to perform a difference capture on each of theterminals 1 a-1 d. More specifically, the difference capture managementflag in the display status management file 34 is set to be “I”.

For a first cycle of process subsequent to the setting of the differencecapture management flag to “1”, steps S1-S18 are performed as describedabove to present the multi-window screen.

After the completion of the multi-window presentation, the arithmeticunit 27 in the display device 2 sends a difference capture start commandtogether with the display status management file 34 to the terminal 1,having the difference capture management flag set to “1”, through thecommunication unit 28 and the network 3.

The operation of the terminal 1 having received the difference capturestart command and the display status management file 34 from the displaydevice 2 will be discussed below.

FIG. 22 is a flow diagram illustrating the operation of terminal whichhas received the difference capture start command and the display statusmanagement file 34 from the display device 2. Here, the terminal is theterminal 1 a (PC-1).

The data management processor 18 in the terminal 1 a receives, throughthe communication unit 17, the difference capture start command and thedisplay status management file 34 sent by the display device 2 (stepS41). The terminal 1 a thus performs a subsequent full-screen captureprocess. More specifically, the screen capture processor 19 stores thecontent of the video memory 12 in the storage 15 (step S42). The imagedata obtained here is referred to as pre-full-screen data.

The screen comparison processor 20 references the difference capturemanagement flag in the display status management file 34, received bythe data management processor 18, at any regular intervals set (onceevery 0.5 second, for example) (step S43). If the difference capturemanagement flag is “1” (step S44), a subsequent difference captureoperation is performed.

To quit the difference capture operation, the user simply sets thedifference capture management flag to “0” in the same way the displaystatus management file 34 is modified. The screen comparison processor20 determines whether or not the capture operation is suspended, byreferencing the difference capture management flag.

After performing the full-screen capture process, the screen captureprocessor 19 stores the content of the video memory 12 in the storage 15(step S45). The image data obtained here is referred to aspost-full-screen data. The screen comparison processor 20 in thecontroller 16 compares the pre-full-screen data with thepost-full-screen data (step S46).

A screen comparison process is discussed in which a mouse pointer ismoved (see FIG. 23). The mouse pointer changes the position thereof frompre-full-screen data 60 and post-full-screen data 61. As shown in FIG.24, two areas 70 and 71 are recognized as being changed. The screencomparison processor 20 detects a change. If it is determined that therehas been a change in the screen (step S47), the screen comparisonprocessor 20 acquires image data of an area that has undergone a change,a size of the image data (the number of pixels in vertical andhorizontal directions), and coordinates of an origin of the image data(absolute coordinates within the area defined by the captured image sizein the display status management file 34).

In this case, the screen comparison processor 20 acquires, with respectto the area 70, captured image data of the area 70 (hereinafter referredto difference captured image data 1), a difference captured image size1, and a difference capture origin 1, and with respect to the area 71,captured image data of the area 71 (hereinafter referred to differencecaptured image data 2), a difference captured image size 2, and adifference capture origin 2. The screen comparison processor 20 thenstores these pieces of information in the storage 15 while notifying thedata management processor 18 that the difference captured image data hasbeen acquired (step S48). The difference captured image size 1, thedifference captured image size 2, the difference capture origin 1, andthe difference capture origin 2 are written on the respective portionsthereof in the display status management file 34.

The data management processor 18 sends these pieces of data to thedisplay device 2 through the communication unit 17 and the network 3(step S49). The difference capture function is different from thefull-screen capture and the partial-screen capture. More specifically,the difference capture function is performed not in response to therequest to send received from, the display device 2 but in response tothe change in the screen detected by the screen comparison processor 20.The transmission operation of the captured image data is then performed.

The post-full-screen data is set to be pre-full-screen data for asubsequent screen comparison process (step S50).

The display device 2 then receives the difference captured image data 1,the difference captured image data 2, and the display status managementfile 34 from the terminal 1 a. The operation of the display device 2subsequent to the reception of these pieces of data will now bediscussed.

Upon receiving the difference captured image data 1, the differencecaptured image data 2, and the display status management file 34 fromthe terminal 1 a at the communication unit 28, the arithmetic unit 27transfers an image synthesis command to the image synthesizer 42. Thedisplay device 2 enters the image synthesis process in the differencecapture function.

Upon receiving the image synthesis command, the image synthesizer 42rewrites portions of the captured image data stored in the capturedimage data memory 37 corresponding to the difference captured image data1 and the difference captured image data 2, based on the window areainformation file 36 corresponding to the image displayed on the display21 and the display status management file 34 (the difference capturedimage size 1, the difference captured image size 2, the differencecapture origin 1, and the difference capture origin 2) received from theterminal 1 a. The process subsequent to this operation are identical tothose in steps S17 and S18.

The amount of image data transmitted over the network 3 is smaller inthe difference capture process than in the full-screen capture (or thepartial-screen capture) process. The workload on the network 3 is thusreduced. The user constantly monitors an updated image of the terminal1.

The above description of the difference capture process is based on theassumption that a dedicated program installed on the terminal 1 is usedto capture the screen. If a driver for directly detecting a differencein the content of the video memory is available on the operating system(OS), such a driver may be used.

The image data may be exchanged in a compression standard format (suchas JPEG) between the terminal 1 and the display device 2 to reduce theworkload on the network 3.

In the above description, a variety of requests such as the displayrequest and the single-window screen presentation request is performedby operating the input section 23 on the display device 2. In otherwords, the display device 2 has the initiative in the organization ofthe screen. Alternatively, each terminal 1 may have the initiative. Theconference participant (the user) enters a desired operational input byoperating the input section 13 on own terminal 1. The input informationis transferred to the data management processor 18 through the userinterface 14. The data management processor 18 in turn sends therequest, including the terminal name and the IP address of own terminal1, responsive to the input information to the display device 2 via thecommunication unit 17 and the network 3. The request is received by thecommunication unit 28 in the display device 2 through the network 3. Therequest is then transferred to the arithmetic unit 27. The operationsubsequent thereto remains the same as those already discussed. In thisway, a variety of requests may be placed on the terminal 1.

The terminal 1 may designate the capture area by containing information,which designates the full-screen capture or the partial-screen capture,in the variety of requests.

The priority order may be designated by the user as necessary asdescribed above. Alternatively, a plurality of terminals 1 which arescheduled to be connected to the display device 2 may be assignedbeforehand priority order. If the terminals 1 are assigned beforehandthe priority order, the priority assigned to each terminal 1 isautomatically set in the priority item of the display status managementfile 34 when the terminal information is registered in the displaystatus management file 34.

In the second preferred embodiment, the tentative window area isdetermined based on the tentative window area setting file 35.Alternatively, the tentative window area may be determined throughcalculation each time.

What is claimed is:
 1. A display method of a display device connected toa plurality of terminals via a network, the display device displayingone or more split window areas, comprising: receiving identificationinformation and a display request of a terminal among the terminals fromthe terminal, determining whether or not the terminal is permitted toconnect to the display device and whether or not the terminal ispermitted to display to the display device, and displaying, when theterminal is a terminal permitted to display, a multi-window screen thatis synthesized based on image data and predetermined informationreceived from the terminal.
 2. The display method of claim 1, whereinthe display device determines whether or not the terminal requesting toconnect to the display device is permitted to connect based on theidentification information thereof.
 3. The display method of claim 1,wherein the display device determines whether or not the terminal ispermitted to display to the display device based on the identificationinformation thereof.
 4. The display method of claim 1, wherein thedisplay device determines a display position in the multi-window screenfor each of the terminals based on such a predetermined information ofthe terminals.
 5. The display method of claim 1, wherein the displaydevice displays a screen while maintaining an aspect ratio of a displayof the terminal.
 6. The display method of claim 1, wherein wheninformation coming from each of the terminals does not include datarelated to a priority among the terminals and screen sizes (resolution)of the terminals are same, the one or more split window areas aredivided into an equal screen size.
 7. The display method of claim 1,wherein when receiving a single-window display request related to aterminal corresponding to one area of the split window areas in themulti-window screen to be displayed on a single-window screen, thedisplay device displays the single-window screen that is switched fromthe multi-window screen.
 8. The display method of claim 7, wherein thesingle screen display request includes identification information of thecorresponding terminal.
 9. The display method of claim 7, wherein thedisplay device displays the multi-window screen that is switched backfrom the single-window screen by a predetermined operation.
 10. Thedisplay method of claim 1, wherein when receiving a new display request,the display device displays the multi-window screen with inserting a newsplit window area.
 11. The display method of claim 1, wherein whenreceiving an erase request related to a terminal corresponding to onearea of the split window areas in the multi-window screen to be erased,the display device displays the multi-window screen with deleting theone area.
 12. The display method of claim 1, wherein the terminaltransmits the image data to the display device after converting size ofscreen data at a terminal side
 13. The display method of claim 1,wherein the terminal permitted for display transmits new image data fromthe terminal to the display device as a trigger for a change in a screenof the terminal and the display device displays with using a differencebetween the displayed image data and the new image data.
 14. The displaymethod of claim 1, wherein each of the terminals has a display, and eachof the terminals obtains, from the display device, a terminal name of aterminal permitted to be connected in a permitted connection managementfile, a terminal name of a terminal permitted to display in a permitteddisplay management file, or a terminal name of a terminal beingconnected to the display device in a connection status management file,and displays the terminal names on the display.
 15. The display methodof claim 1, wherein each of the terminals has a display, and each of theterminals obtains, from the display device, an information, which is ina display state management file, related to a source terminal displayedon a multi-window screen of the display device from the display device,and displays the information on the display.
 16. The display method ofclaim 15, wherein one of the terminals displays which terminal exceptthe one terminal is displayed in the multi-window screen, on the displayof the one terminal, the one terminal being operated.
 17. A displaysystem comprising: a plurality of terminals that generate a plurality ofimages; and a display device that projects the plurality of imagestransmitted from the terminals, wherein the display device includes: afirst display that receives the images generated by the terminals fromthe terminals and displays the images; a controller that generatesinformation of a connection state with the terminals or a displaystatus; and a first communication unit that transmits the generatedinformation to the terminals, and each of the terminals includes: asecond communication unit that receives the information from the displaydevice; and a second display that displays whether or not other terminalof the terminals is in connection with the display device, based on theinformation.
 18. A display system comprising: a plurality of terminalsthat generate a plurality of images; and a display device that projectsthe plurality of images transmitted from the terminals, wherein thedisplay device includes: a first display that receives the imagesgenerated by the terminals from the terminals and displays the images; acontroller that generates information of a connection state with theterminals or a display status; and a first communication unit thattransmits the generated information to the terminals, and each of theterminals includes: a second communication unit that receives theinformation from the display device; and a second display that displayswhether or not other terminal of the terminals is displayed in the firstdisplay of the display device, based on the information.